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EXECUTIVE  SUMMARY 

Assignment  of  personnel  to  a  language  for  initial  acquisition  training  is  an  important  decision  with  many 
potential  implications.  It  is  important  to  assign  individuals  to  languages  in  which  there  is  high  likelihood 
that  they  will  succeed.  If  individuals  are  not  correctly  assigned,  this  can  have  financial,  capability,  and 
mission  performance  consequences.  To  assign  individuals  to  language  training,  the  United  States  military, 
including  the  SOF  community,  relies  on  the  Defense  Language  Aptitude  Battery  (DLAB).  Studies  within 
the  Department  of  Defense  (DoD)  have  demonstrated  the  reliability  (i.e.,  consistency)  and  predictive 
validity  (i.e.,  predicting  training  outcomes)  of  the  DLAB  (Petersen  &  Al-Haik,  1976;  Silva  &  White, 
1993);  however,  studies  within  the  SOF  community  found  lower  predictability  (SWA  Consulting, 
Technical  Report  #2009010616,  Technical  Report  #2009010612;  see  Section  II  for  further  explanation). 
Statistical  reliability  and  validity  are  important;  however,  if  a  test  is  not  perceived  to  be  reliable  and  valid, 
these  perceptions  could  lead  to  negative  test-taking  behaviors  and  eventually  undermine  the  test’s 
statistical  properties.  It  is  equally  important  that  test  takers  perceive  that  the  test  is  accurate  and  fair  to 
ensure  that  the  test  and  its  results  are  accepted  and  that  individuals  continue  to  engage  in  the  testing 
process  (e.g.,  not  undermining  it  through  lack  of  effort). 

Perceived  test  accuracy  (at  face  value,  is  the  test  measuring  what  it’s  supposed  to  be  measuring?;  face 
validity)  is  positively  related  to  test  motivation  and  attitudes  toward  tests  (Hausknecht,  Day,  &  Thomas, 
2004).  If  individuals  perceive  the  test  as  accurate  prior  to  taking  the  test,  then  they  are  more  likely  to  take 
the  test  seriously  and  put  forth  effort  during  test  administration,  preserving  the  test’s  psychometric 
properties  (e.g.,  predictive  validity  for  post-training  proficiency).  Additionally,  research  has  shown  that 
test  fairness  perceptions  are  related  to  factors  other  than  how  the  individual  performs  on  the  test  (Bauer, 
Maertz,  Dolen,  &  Campion,  1998).  These  factors,  called  procedural  justice  factors,  can  influence  test 
accuracy  and  fairness  perceptions.  Examples  of  procedural  justice  factors  include  having  prior  knowledge 
about  the  test  before  testing,  having  a  chance  to  demonstrate  skills  on  the  test,  and  treatment  experienced 
at  the  testing  site. 

To  determine  if  the  DLAB  functions  and  is  used  as  intended,  the  initial  step  is  to  assess  perceptions  of  test 
accuracy  and  fairness.  If  findings  indicate  that  the  DLAB  is  perceived  as  inaccurate  and/or  unfair,  it  is 
important  to  identify  procedural  justice  factors  that  may  be  contributing  to  these  perceptions.  These 
factors  can  identify  opportunities  for  improvement  in  the  use  of  the  DLAB,  optimizing  its  effectiveness 
for  language  assignment.  The  2009  SOF  Language  and  Culture  Needs  Assessment  Project  (LCNA) 
assessed,  1)  perceptions  of  the  DLAB’s  accuracy  in  measuring  language  learning  aptitude  (i.e.,  the  test’s 
perceived  accuracy  or  test’s  face  value),  2)  perceptions  about  the  DLAB’s  use  for  assigning  individuals  to 
languages  (i.e.,  test  fairness  perceptions),  and  3)  potential  factors  that  could  influence  SOF  personnel’s 
perception  of  DLAB  accuracy  and  fairness. 

Is  the  DLAB  an  accurate  measure  of  language  learning  aptitude? 

SOF  personnel  had  mixed  opinions  regarding  the  DLAB’s  accuracy  in  measuring  language  learning 
aptitude.  Personnel  currently  in  the  training  pipeline  (62%,  n  =  50)  and  SOF  leaders  (55%,  n  =  51)  more 
frequently  indicated  that  the  DLAB  accurately  reflects  language  learning  aptitude  than  SOF  operators 
(46%,  n  =  411). 


09/29/10 


©  SWA  Consulting  Inc.,  2010 
Technical  Report  [201001 1017] 


Page  2 


SOF  Language  and  Culture  Needs  Assessment  Project 


Defense  Language  Aptitude  Battery  (DLAB) 


Should  it  be  used  to  assign  individuals  to  language? 

SOF  personnel  also  had  mixed  opinions  regarding  whether  or  not  the  DLAB  should  be  used  to  assign 
individuals  to  language  training.  Sixty-two  percent  (w  =  51)  of  personnel  currently  in  the  training  pipeline 
and  71%  (n  =  65)  of  leaders  indicated  that  the  DLAB  should  be  used  to  assign  individuals  to  a  language, 
compared  to  only  49%  (n  =  43 1)  of  operators.  The  higher  percentage  of  leaders  and  current  trainees 
indicating  that  the  DLAB  should  be  used  for  language  assignment  may  be  explained  by  their  relative 
familiarity  with  how  the  test  scores  are  used  or  should  be  used  in  the  assignment  process  due  to  their 
current  roles  in  the  organization. 

Based  on  whether  or  not  the  test  is  accurate,  should  it  be  used  to  assign  individuals  to  language? 

There  were  four  groups  of  operator  respondents: 

•  DLAB  is  not  accurate  and  should  not  be  used  to  assign  individuals  to  language  training  (43%,  n  = 
377) 

•  DLAB  is  accurate  and  it  should  be  used  to  assign  individuals  to  language  training  (38%,  n  =  330) 

•  DLAB  is  not  an  accurate  reflection  of  their  language  learning  aptitude  but  it  should  be  used  to 
assign  individuals  to  language  training  (10%,  n  =  90) 

•  DLAB  is  an  accurate  reflection  but  should  not  be  used  to  assign  individuals  to  language  training 
(8%,  n  =  73) 

Respondents  from  the  first  two  groups  provided  comments  referencing  their  own  experience  with  DLAB 
scores  and  subsequent  language  learning  or  performance  (e.g.,  DLAB  score  matched  their  performance  in 
a  language;  DLAB  score  did  not  match  their  performance  in  a  language).  Comments  from  those  who  said 
the  test  is  accurate  but  should  not  be  used  for  language  assignment  suggest  that  other  factors  should  be 
taken  into  account  in  the  language  assignment  process  in  combination  with  DLAB  scores,  including 
personal  motivation  to  learn  a  particular  language  or  previous  experience  in  learning  a  language. 
Comments  provided  by  respondents  who  indicated  that  the  test  is  not  accurate  but  should  be  used  to 
assign  revealed  the  perspective  that  the  DLAB  is  the  best  (or  only)  available  language  learning  aptitude 
measure. 

What  factors  influence  SOF  personnel ’s  perceptions  of  accuracy  and  fairness  of  the  DLAB? 

Given  that  some  SOF  personnel  perceived  the  DLAB  as  inaccurate  and/or  unfair  to  use  for  language 
assignment,  it  is  important  to  assess  test  taker  reactions  to  determine  why  individuals  hold  these 
perceptions.  Survey  comments  and  focus  group  discussions  revealed  factors  influencing  the  DLAB’s 
perceived  accuracy  and  fairness  (see  Section  IV): 

•  No  knowledge  of  what  the  DLAB  measures  or  how  it  is  used  in  the  language  assignment  process 
prior  to  testing 

•  Perceptions  that  the  DLAB  does  not  measure  all  aspects  of  language  learning  aptitude 

•  Perceptions  that  the  DLAB  measures  skills  unrelated  to  language  learning  (i.e.,  test- taking  skills) 

•  Perceptions  or  experiences  that  suggest  DLAB  scores  are  not  used  to  assign  individuals  to 
language 

•  Negative  testing  experiences  (e.g.,  testing  conditions,  administrative  mistakes) 
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These  factors  align  with  factors  identified  by  past  research  (Bauer  et  al.,  1998;  Bauer  et  al.,  2001; 

Gilliand,  1993),  including  information  known  about  the  test  prior  to  testing,  the  test  providing  an 
adequate  opportunity  to  demonstrate  skills  and/or  knowledge,  and  conditions  and/or  treatment  at  the 
testing  site.  These  reactions  contribute  to  test  accuracy  and  fairness  perceptions  (Bauer  et  al.,  1998). 

These  factors  should  be  addressed  to  improve  operator  and  leader  perceptions.  Some  of  these  factors  are 
characteristics  of  the  test  and  can  only  be  addressed  by  improving  or  replacing  the  DLAB;  others  can  be 
addressed  through  communication  and  consistent  use  and  administration  of  the  DLAB.  SOF  leaders  can 
advocate  for  changes  to  the  DLAB  and/or  seek  alternatives,  but  leaders  who  control  the  DLAB 
administration  and  the  use  of  scores  for  language  assignment  can  achieve  the  most  direct  impact.  These 
leaders  can  ensure  effective  administration  and  use  of  the  DLAB,  optimizing  the  predictive  value  of  the 
test  and  improving  resulting  language  assignments.  The  following  recommendations  provide  ways  to 
reduce  or  eliminate  these  factors. 

First,  all  personnel  who  take  the  DLAB  should  be  educated  about  what  the  DLAB  measures  and  how  it  is 
used  in  language  assignment  prior  to  taking  the  test,  especially  if  this  can  be  done  in  such  a  way  as  to  not 
compromise  the  DLAB’s  functioning.  As  described  by  Bauer  and  colleagues  (1998,  2001),  information 
known  about  the  test  contributes  to  an  individual’s  perception  of  whether  or  not  it  is  credible  and  fair.  If 
an  individual  does  not  know  what  the  test  intends  to  measure  or  how  it  will  be  used,  then  their  perceptions 
of  the  test’s  accuracy  and  fairness  are  likely  to  be  negative.  Survey  comments  indicated  that  some 
operators  did  not  know  what  the  test  was  supposed  to  measure  prior  to  testing.  These  negative  perceptions 
may  influence  their  motivation  to  perform  well  on  the  test.  It  should  be  noted  that  explaining  to  test  takers 
what  the  DLAB  intends  to  measure  and  how  it  will  be  used  in  the  language  assignment  process  can  lead 
to  some  test  takers  intentionally  underperforming  on  the  test  as  a  strategy  to  avoid  being  assigned  to  a 
more  difficult  language.  In  this  case,  presenting  the  DLAB  as  one  factor  in  the  language  assignment 
process  might  be  most  beneficial.  One  alternative  strategy  is  to  inform  test  takers  of  the  purpose  and  use 
of  the  DLAB  after  they  complete  the  test  (i.e.,  debriefing).  The  debriefing  strategy  might  improve 
perceptions  of  the  test  but  does  not  address  the  pre-DLAB  motivation  issue.  Also,  the  DLAB  is  often 
administered  prior  to  individuals  entering  the  SOF  training  pipeline  or  the  SOF  community.  In  this  case,  it 
is  impossible  to  communicate  the  use  of  the  DLAB  in  SOF  language  assignment  or  the  importance  of  the 
test.  However,  this  communication  is  possible  when  the  DLAB  is  administered  as  part  of  the  process  in  a 
SOF  organization.  Regardless,  test  takers  need  some  information  prior  to  testing  to  inform  their 
motivation  and  engagement  in  the  DLAB  testing  process. 

Second,  SOF  personnel’s  perceptions  about  whether  the  DLAB  measures  all  aspects  of  language  learning 
aptitude  and  whether  it  measures  abilities  unrelated  to  language  learning  should  be  monitored.  As 
indicated  by  Bauer  and  colleagues  (2001),  having  a  chance  to  demonstrate  knowledge,  skills,  and  abilities 
on  the  test  contributes  to  perceptions  of  its  accuracy  and  fairness.  If  the  test  does  not  measure  all  aspects 
of  what  it  is  supposed  to  measure  or  if  the  test  measures  something  unrelated  to  what  it  should  be  testing, 
then  the  test  may  be  perceived  negatively.  For  example,  some  operators  reported  that  the  DLAB  does  not 
measure  all  aspects  of  language  learning  aptitude  (e.g.,  motivation  or  personal  interest  in  a  particular 
language),  and  some  operators  suggested  that  the  DLAB  measures  aspects  that  are  unrelated  to  the  ability 
to  learn  a  foreign  language  (i.e.,  test  taking  skills).  These  negative  perceptions  can  be  passed  to 
individuals  who  have  not  yet  taken  the  test,  negatively  influencing  their  motivation  to  engage  in  the 
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DLAB  testing  process  and  their  resulting  scores.  The  less  accurate  scores  resulting  from  unmotivated  test 
performance  may  undermine  the  statistical  properties  of  the  scores  and  the  resulting  language 
assignments.  Although  the  DLAB  testing  program  is  controlled  outside  of  USSOCOM,  USSOCOM 
should  communicate  its  testing  requirements  and  feedback  to  the  appropriate  groups.  An  effective 
language  aptitude  test  should  not  only  have  sufficient  psychometric  properties  (validity  and  reliability) 
and  predictive  validity  for  SOF  language  training  outcomes,  but  it  should  also  include  relevant  factors  and 
exclude  irrelevant  factors.  Users  should  perceive  the  test  as  being  an  accurate  predictor  to  maximize  the 
effectiveness  of  its  use. 

Third,  to  preserve  or  increase  perceived  test  accuracy  and  fairness,  DLAB  scores  should  be  used 
consistently  to  assign  individuals  to  language  training.  As  indicated  by  Bauer  and  colleagues  (2001),  test 
administration  and  decision  procedure  consistency  influence  test  perceptions.  If  the  test  is  not  consistently 
used  for  language  assignment,  then  personnel  may  not  take  the  test  seriously  because  they  anticipate  that 
their  test  score  will  not  be  used  in  the  assignment  process.  Some  respondents  indicated  that  the  DLAB 
should  not  be  used  for  language  assignment  because  of  credibility  issues.  If  factors  that  negatively 
influence  DLAB  accuracy  perceptions  can  be  addressed  then  the  fairness  perceptions  of  the  language 
assignment  process  should  improve.  Additionally,  test  takers  may  not  perceive  the  language  assignment 
process  as  fair  because  their  DLAB  scores  are  not  used  as  originally  intended.  If  other  factors  (e.g., 
previous  language  experience,  current  and  future  force  requirements)  are  considered  in  the  language 
assignment  process  or  can  override  the  DLAB  in  the  assignment  decision,  the  full  assignment  criteria  and 
process  should  be  communicated  to  personnel  at  the  appropriate  time  in  the  process. 

Finally,  test  takers  should  be  provided  with  reasonable  testing  accommodations.  Conditions  and  treatment 
at  the  testing  site  can  influence  whether  test  takers  perceive  the  test  as  credible  and  fair  (Bauer  et  al., 
2001).  For  example,  some  operators  reported  that  the  temperature  of  their  testing  environment  was  too 
hot.  Furthermore,  some  operators  reported  they  did  not  receive  adequate  sleep  the  night  before  the  test 
due  to  other  training  requirements.  These  factors  can  systematically  impact  individual  test  performance  in 
groups  that  experience  these  constraints  and,  therefore,  alter  or  undermine  predictive  validity.  This  can 
lead  to  inaccurate  assignments  and  diminished  credibility  and  fairness  perceptions  of  the  DLAB.  If  these 
administrative  constraints  are  expected  to  occur  in  the  future,  research  should  be  conducted  under  these 
constrained  conditions  to  determine  the  impact  on  the  predictive  validity  of  the  DLAB.  Lastly,  some 
operators  reported  test  administration  issues  (e.g.,  test  administrator  not  taking  the  test  seriously  or  not 
providing  clear  test  instructions).  All  test  administrators  should  be  trained  to  provide  adequate  testing 
instructions  for  test  takers  and  maintain  a  neutral  attitude  about  the  test. 

Although  perceptions  of  the  DLAB’s  accuracy  and  its  use  for  language  assignment  were  mixed,  steps  can 
be  taken  to  improve  perceptions  of  the  DLAB.  First,  communicating  the  DLAB  testing  policy  can  educate 
personnel  about  when  the  test  is  typically  administered,  what  the  test  measures,  and  how  it  is  used  in  the 
SOF  language  assignment  process.  Second,  encouragement  of  effective  and  consistent  use  of  the  DLAB 
for  language  assignment  can  assist  in  making  sure  that  the  test  scores  are  used  as  described  by  the  DLAB 
testing  policy.  If  a  test  is  effectively  and  consistently  used,  perceptions  of  the  test’s  accuracy  and  fairness 
may  improve.  This  leads  to  test  takers  engaging  in  the  testing  process  and  results  in  their  test  scores  being 
accurate  and  representative  of  their  language  learning  aptitude.  Using  accurate  and  representative  scores 
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for  language  assignment  leads  to  appropriate  assignments  and  can  provide  the  organization,  the  unit,  and 
the  individual  with  the  best  opportunity  to  succeed  in  the  language  domain. 

Future  Directions 

The  limited  predictive  validity  of  the  DLAB  and  its  age  as  an  assessment  led  to  funding  for  development 
of  the  DLAB  2.  University  of  Maryland’s  Center  for  Advanced  Study  of  Language  (CASL)  is  developing 
a  new  version  of  the  DLAB  (the  DLAB  2)  to  incorporate  new  cognitive  and  non-cognitive  measures  (e.g., 
personality  and  motivation)  that  may  predict  foreign  language  aptitude.  This  new  version  should  address 
some  of  the  original  DLAB’s  shortcomings  which  were  identified  by  respondents  in  this  study,  such  as 
the  lack  of  motivation  measures.  If  this  test  is  eventually  used  by  USSOCOM  for  language  placement, 
then  the  DLAB  2’s  incorporation  of  motivation  measures  into  the  testing  process  may  lead  SOF  personnel 
to  perceive  the  test  and  language  assignment  process  as  more  credible  and  fair.  When  the  DLAB  2  is 
available,  it  should  be  validated  for  use  with  SOF  personnel,  if  the  test  is  feasible  for  administration 
within  the  constraints  of  SOF  operational  use.  The  initial  validation  report  for  the  DLAB  2  is  expected  to 
be  released  in  late  2010  and  will  accompany  two  other  pilot  studies.  These  reports  were  not  available  at 
the  time  of  this  report’s  completion. 

See  Appendix  A  of  this  report  for  additional  details  about  the  SOF  LCNA  Project.  For  questions  or  more 
information  about  the  Special  Operations  Forces  Language  Office  (SOFLO)  and  this  project,  please 
contact  Mr.  Jack  Donnelly  (iohn.donnelly@socom.mil).  For  specific  questions  related  to  data  collection 
or  reports  associated  with  this  project,  please  contact  Dr.  Eric  A.  Surface  (esurface@swa-consulting.com) 
or  Dr.  Reanna  Poncheri  Harman  (rpharman@swa-consulting.com)  with  SWA  Consulting  Inc. 
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SECTION  I:  REPORT  AND  PROJECT  OVERVIEW 
DLAB:  Perspectives  from  the  Field  Report  Purpose 

The  Defense  Language  Aptitude  Battery  (DLAB)  is  used  by  the  United  States  military  to  assign 
individuals  to  language  training,  such  that  lower  scoring  individuals  are  placed  into  languages  that  are  less 
difficult  for  native  English  speakers  to  learn  (e.g.,  Spanish,  French,  Italian,  German)  and  higher  scoring 
individuals  are  placed  into  more  difficult  languages  (e.g.,  Arabic,  Chinese,  Korean,  Russian).  In  the 
Special  Operations  Forces  (SOF)  community,  other  factors  are  also  considered  in  the  assignment  process, 
including  prior  experience  in  a  language  as  well  as  current  and  future  force  requirements.  Studies  have 
demonstrated  adequate  predictive  validity  for  the  DFAB  predicting  post-training  language  proficiency 
within  the  Department  of  Defense  (DoD)  community  (e.g.,  Silva  &  White,  1993),  but  less  so  in  the  SOF 
community  (SWA  Consulting,  Technical  Report  #2009010616,  Technical  Report  #2009010612;  see 
Section  II).  Although  a  test’s  statistical  properties  are  important,  test  taker  perceptions  of  test  reliability 
and  validity  are  equally  important  to  ensure  that  the  test  and  its  results  are  accepted  and  that  individuals 
continue  to  engage  in  the  testing  process  (e.g.,  not  undermining  it  through  lack  of  effort).  To  assess  test 
taker  perceptions  of  the  DFAB,  this  study  measured  SOF  personnel’s  perceptions  of  test  accuracy  (at  face 
value,  is  the  test  measuring  what  it’s  supposed  to  be  measuring?;  face  validity)  and  test  use  (i.e.,  should 
the  test  be  used  for  language  assignment?). 

Research  has  demonstrated  that  perceived  test  accuracy  influences  attitudes  toward  tests  and  test  fairness 
perceptions  (Hausknecht,  Day,  &  Thomas,  2004).  This  suggests  that  any  preconceived  notion  that  test 
takers  have  regarding  a  test’s  accuracy  can  influence  how  they  approach  taking  the  test.  If  individuals 
perceive  the  test  as  inaccurate  then  they  may  not  take  the  test  seriously,  thereby  undermining  the  tests’ 
predictive  qualities  because  their  scores  do  not  reflect  their  true  ability.  Furthermore,  perceptions  of  the 
test  may  be  communicated  to  others  who  have  not  yet  taken  the  test,  which  could  negatively  influence 
their  attitudes  toward  the  test  and  how  they  engage  in  the  testing  process.  Therefore,  it  is  important  to 
assess  test  taker  perceptions  of  the  DFAB  to  determine  if  SOF  personnel  perceive  the  DFAB  as  an 
accurate  or  inaccurate  measure  of  language  learning  aptitude.  Perceived  test  accuracy  also  influences 
perceptions  of  test  fairness  and  fairness  about  how  test  scores  are  used  (Bauer,  Maertz,  Dolen,  & 

Campion,  1998;  Bauer  et  al.,  2001;  Gilliand,  1993).  If  individuals  perceive  the  test  as  inaccurate,  then 
they  will  likely  perceive  the  test  and  its  use  in  a  selection  or  assignment  context  as  unfair. 

Research  has  also  shown  that  test  accuracy  and  fairness  perceptions  are  influenced  by  factors  other  than 
how  the  individual  performs  on  the  test  (Bauer  et  al.,  1998).  Factors  that  contribute  to  fairness  perceptions 
include  having  prior  knowledge  about  the  test  before  testing,  having  a  chance  to  demonstrate  skills  on  the 
test,  and  conditions  or  treatment  experienced  at  the  testing  site.  For  individuals  who  may  perceive  the  test 
as  inaccurate  and/or  unfair  to  use  for  language  assignment,  these  factors  may  contribute  to  their 
perceptions  and  should  be  identified  and  addressed.  This  suggests  a  diagnostic  approach  and  the 
collection  of  test  taker  feedback. 

To  determine  if  test  takers  believe  the  DFAB  functions  and  is  used  as  intended,  the  initial  step  is  to  assess 
test  taker  perceptions  of  its  accuracy  and  fairness.  If  findings  indicate  that  the  DFAB  is  perceived  as 
inaccurate  and/or  unfair,  it  is  important  to  identify  factors  that  may  be  contributing  to  these  perceptions. 
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This  report  focuses  on  SOF  personnel’s  perceptions  of  the  DLAB’s  accuracy  and  its  use  for  assigning 
individuals  to  language  training  (i.e.,  test  fairness  perceptions).  Factors  that  influence  SOF  personnel’s 
perception  of  DLAB  accuracy  and  fairness  are  explored  as  well. 

This  report  presents  the  specific  details  and  recommendations  related  to  main  findings  identified  by  SOF 
operators  and  leaders  who  participated.  The  report  is  divided  into  five  sections  with  a  number  of 
supporting  appendices.  Section  II  of  this  report  provides  background  on  the  DLAB,  studies  related  to  its 
predictability,  and  preliminary  information  about  the  DLAB  2.  Section  III  of  this  report  addresses  the 
questions,  do  SOF  personnel  perceive  the  DLAB  as  an  accurate  measure  of  language  learning  ability  and 
do  they  believe  it  should  be  used  to  assign  individuals  to  language?  Section  IV  summarizes  SOF  operator 
and  leader  comments  about  the  DLAB  and  identifies  factors  influencing  the  DLAB’s  perceived  credibility 
in  the  SOF  community.  Section  V  concludes  the  report  by  integrating  main  findings  from  each  section 
and  providing  implications  and  recommendations  for  action.  The  References  section  presents  the  research 
and  other  documents  cited  in  this  report.  Appendix  A  details  the  2009  SOF  Language  and  Culture  Needs 
Assessment  Project  (LCNA  Project),  and  Appendix  B  provides  an  overview  of  report  methodology, 
including  participants,  measures,  and  analyses.  Appendix  C  includes  survey  comment  themes,  definitions, 
and  examples. 

LCNA  Project  Purpose 

The  Special  Operations  Forces  Language  Office  (SOFLO)  commissioned  the  2009  SOF  LCNA  Project  to 
gain  insights  on  language  and  culture  capability  and  issues  across  the  United  States  Special  Operations 
Command  (USSOCOM).  The  goal  of  this  organizational-level  needs  assessment  is  to  inform  strategy  and 
policy  to  ensure  SOF  personnel  have  the  language  and  culture  skills  needed  to  conduct  their  missions 
effectively.  Data  were  collected  between  March  and  November,  2009  from  personnel  in  the  SOF 
community,  including  operators  and  leaders.  Findings,  gathered  via  focus  groups  and  a  web-based  survey, 
will  be  presented  in  a  series  of  reports  divided  into  three  tiers.  The  specific  reports  in  each  of  these  tiers 
will  be  determined  and  contracted  by  the  SOFLO.  Tier  I  reports  focus  on  specific,  limited  issues  (e.g., 
Inside  AOR  Use  of  Language).  Tier  II  reports  integrate  and  present  the  most  important  findings  across 
related  Tier  I  reports  (e.g.,  Use  of  Language  and  Culture  on  Deployment)  while  including  additional  data 
and  analysis  on  the  topic.  One  Tier  III  report  presents  the  most  important  findings,  implications,  and 
recommendations  across  all  topics  explored  in  this  project.  The  remaining  Tier  III  reports  present  findings 
for  specific  SOF  organizations  [e.g.,  Air  Force  Special  Operations  Command  (AFSOC),  Special  Forces 
(SF)  Command].  Two  foundational  reports  document  the  methodology  and  participants  associated  with 
this  project.  Report  topics  are  determined  by  the  SOFLO  and  are  subject  to  change. 

Relationship  of  DLAB:  Perspectives  from  the  Field  to  the  LCNA  Project 

The  DLAB:  Perspectives  from  the  Field  report  is  a  Tier  I  report  that  will  be  integrated  with  other  Tier  I 
reports,  Defense  Language  Proficiency  Test  (DLPT)  and  Oral  Proficiency  Interview  (OPI),  into  a  Tier  II 
report,  Testing/Metrics  (see  Appendix  A  for  the  report  structure).  However,  the  final  reports  produced 
will  be  determined  by  the  SOFLO  and  are  subject  to  change. 
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SECTION  II:  DLAB  BACKGROUND 
What  is  the  DLAB  and  what  does  it  measure? 

The  Defense  Language  Aptitude  Battery  (DLAB)  is  a  test  that  measures  ability  to  learn  a  foreign 
language.  DLAB  scores  can  be  used  to  predict  the  likelihood  of  an  individual  learning  a  foreign  language 
within  a  structured  training  program.  It  utilizes  an  artificial  language  to  measure  three  different  abilities 
needed  to  learn  and  understand  a  foreign  language:  1)  processing  of  auditory  phonetic  material  for 
recognition  and  recall,  2)  grammatical  sensitivity,  and  3)  capability  to  learn  new  associations  (Silva  & 
White,  1993).  The  DLAB  items  ask  test  takers  to  identify  an  accurate  translation  by  applying  artificial 
language  grammar  rules,  matching  pictures  to  phrases,  and  recognizing  vowel  stress  patterns. 

The  DLAB  has  been  used  at  the  Defense  Language  Institute  (DLI)  since  the  1970s  and  is  used  by  other 
organizations  in  the  Department  of  Defense  (DoD),  including  the  SOF  community,  for  language  training 
assignment  (i.e.,  higher  scoring  personnel  are  placed  into  more  difficult  languages  and  lower  scoring 
individuals  are  placed  into  less  difficult  languages).  Studies  within  the  DoD  have  demonstrated  the 
DLAB’s  predictive  validity  (i.e.,  predicting  post-training  proficiency  scores  using  DLAB  scores;  Silva  & 
White,  1993)  and  construct  validity  (i.e.,  comparing  the  DLAB  to  other  language  aptitude  tests;  Petersen 
&  Al-Haik,  1976).  Additionally,  the  DLAB  contributed  significant  incremental  (i.e.,  additional)  predictive 
validity  beyond  general  aptitude  (g)  and  beyond  the  Army  Services  Vocational  Aptitude  Battery 
(ASVAB;  Silva  &  White,  1993).  However,  studies  within  the  SOF  community  (SWA  Consulting, 
Technical  Report  #2009010616;  SWA  Consulting,  Technical  Report  #2009010612)  have  found  weaker 
predictability  partially  due  the  majority  of  SOF  operators  scoring  at  the  same  proficiency  level  (i.e.,  1/1) 
after  initial  acquisition  training,  whereas  the  DoD  community  has  a  wider  range  of  proficiency  scores 
(e.g.,  0  through  5  ILR).  Range  restriction  undermines  the  predictability  of  DLAB  scores  for  end-of- 
training  language  proficiency  in  the  SOF  community  because  most  proficiency  results  are  similar, 
therefore,  a  wide  range  of  DLAB  scores  do  not  differentiate  proficiency  outcomes. 

Despite  its  modest  predictability,  studies  within  the  SOF  community  suggest  that  the  DLAB  is  the  most 
consistently  predictive  single  indicator  of  language  proficiency  as  measured  by  the  Defense  Language 
Proficiency  Test  (DLPT)  and  Oral  Proficiency  Interview  (OPI).  These  studies  compared  the  DLAB  to 
alternatives,  such  as  the  Armed  Forces  Qualification  Test  (AFQT)  and  the  Wonderlic  Personnel  Test 
(WPT)1,  to  determine  if  other  tests  or  combinations  of  tests  could  predict  language  proficiency  measured 
by  both  the  DLPT  and  the  OPI.  One  study  found  that  an  AFQT-WPT  composite  produced  DLPT  rating 
prediction  comparable  to  the  DLAB  (SWA  Consulting,  Technical  Report  #2009010616).  Another  study 
found  that  the  DLAB  was  the  best  predictor  of  OPI  ratings  (speaking  and  listening)  when  compared  to  the 
AFQT  and  WPT  (SWA  Consulting,  Technical  Report  #2009010612).  The  limited  predictive  validity  of 
the  DLAB  and  its  age  as  an  assessment  led  to  funding  for  the  development  of  the  DLAB  2. 


1  This  test  is  now  referred  to  as  the  Wonderlic  Classic  Cognitive  Ability  Test  by  the  test  developers;  however,  the  test  will  be 
referred  to  as  the  Wonderlic  Personnel  Test  (WPT)  in  this  report. 
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How  is  the  DLAB  used  in  the  SOF  community  to  assign  individuals  to  languages? 

An  operator’s  DLAB  score  determines  the  difficulty  level  of  the  language  they  will  be  assigned.  Category 
I  (e.g.,  Spanish,  French,  Italian)  and  Category  II  (e.g.,  German)  languages  share  commonalities  with  the 
English  language  and,  therefore,  are  considered  less  difficult  for  native  English  speakers  to  learn  than 
Category  III  (e.g.,  Russian,  Korean)  and  Category  IV  (e.g.,  Arabic,  Chinese)  languages.  According  to 
policy,  the  DLAB  score  should  be  used  to  determine  into  which  category  of  language  a  trainee  is 
assigned.  SOF  cutoffs  for  each  language  category  are  provided  below  (from  USSOCOM  M350-8,  2009): 

•  Assignment  to  a  Category  I  language  requires  minimum  DLAB  score  of  75 

•  Assignment  to  a  Category  II  language  requires  minimum  DLAB  score  of  75 

•  Assignment  to  a  Category  III  language  requires  minimum  DLAB  score  of  80 

•  Assignment  to  a  Category  IV  language  requires  minimum  DLAB  score  of  85 

Some  components  may  require  higher  minimum  scores  than  those  listed  above.  Additionally,  some 
individuals  may  be  waived  into  more  difficult  languages  if  they  do  not  reach  the  DLAB  score  minimum 
for  that  category.  However,  most  language  assignments  are  based  on  the  DLAB  cutoff  scores  listed  above 
with  consideration  of  individual’s  native/heritage  skills  and  prior  language  training/  instruction 
(USSOCOM  M  350-8).  Additionally,  current  and  future  military  needs  are  also  taken  into  account  when 
assigning  individuals  to  language  training.  As  a  point  of  comparison,  the  Department  of  Defense  (DoD) 
cutoffs  for  the  DLAB  are  listed  below  (from  R1 1-6  Army  Linguist  Management): 

•  Assignment  to  a  Category  I  language  requires  minimum  DLAB  score  of  85 

•  Assignment  to  a  Category  II  language  requires  minimum  DLAB  score  of  90 

•  Assignment  to  a  Category  III  language  requires  minimum  DLAB  score  of  95 

•  Assignment  to  a  Category  IV  language  requires  minimum  DLAB  score  of  100 

What  is  the  DLAB  2  and  how  is  it  different  from  the  original  DLAB? 

University  of  Maryland’s  Center  for  Advanced  Study  of  Language  (CASL)  is  developing  a  new  version 
of  the  DLAB  to  incoiporate  new  cognitive  and  non-cognitive  measures  (e.g.,  personality  and  motivation) 
that  may  predict  foreign  language  aptitude.  Additionally,  the  DLAB  2  eliminates  redundant  measures. 

The  initial  validation  report  is  expected  to  be  released  in  late  2010  and  will  accompany  two  other  pilot 
studies.  The  CASL  reports  were  not  available  at  the  time  of  this  report’s  completion. 
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SECTION  III:  DLAB  ACCURACY  AND  ITS  USE  FOR  LANGUAGE  ASSIGNMENT 

Perceived  test  accuracy  (at  face  value,  is  the  test  measuring  what  it  is  supposed  to  be  measuring;  face 
validity)  is  positively  related  to  test  motivation  and  attitudes  toward  tests  (Hausknecht  et  al.,  2004).  If 
operators  and/or  leaders  do  not  perceive  the  DLAB  as  accurate,  then  the  test’s  credibility  is  at  stake.  This 
can  lead  to  negative  test  taking  behaviors  and  negatively  affect  test  taker  motivation,  both  of  which  can 
diminish  the  test’s  predictability.  If  test  takers  perceive  a  test  as  accurate,  they  may  apply  more  effort  in 
taking  the  test  and  take  it  more  seriously  than  test  takers  who  perceive  the  test  as  lacking  credibility. 

In  addition  to  the  DLAB’s  accuracy,  it  is  important  to  capture  the  opinions  of  operators  and  leaders  about 
using  DLAB  scores  to  assign  individuals  to  language  training,  such  that  individuals  with  higher  DLAB 
scores  are  placed  into  more  difficult  languages.  Research  shows  that  test  fairness  perceptions  are  related 
to  factors  other  than  how  the  individual  performs  on  the  test,  such  as  test  administration  and  decision 
procedure  consistency,  conditions  or  treatment  at  the  testing  site,  and  having  a  chance  to  demonstrate 
knowledge,  skills,  and  abilities  on  the  test  (Bauer  et  al.,  2001). 

This  section  presents  SOF  operator  and  leader  perspectives  on  whether  or  not  the  DLAB  is  an  accurate 
reflection  of  language  learning  aptitude  and  whether  or  not  it  should  be  used  to  assign  individuals  to  a 
language.  Opinions  from  the  leader  perspective  provide  a  policy-driven  view  of  using  DLAB  scores  for 
language  assignment,  whereas  the  operator  perspective  provides  insight  into  whether  or  not  the  language 
assignment  process  is  considered  fair  and  justified  by  actual  test  takers. 

Research  Questions 

This  section  addresses  the  following  questions: 

•  Do  operators  and  leaders  believe  that  the  DLAB  is  an  accurate  measure  of  language  learning 
aptitude? 

•  Do  operators  and  leaders  believe  that  the  DLAB  should  be  used  to  assign  individuals  to 
languages? 

Main  Findings 

SOF  personnel  had  mixed  opinions  on  the  DLAB’s  accuracy  in  measuring  language  learning  aptitude  and 
its  use  for  language  assignment.  Overall,  personnel  currently  in  the  training  pipeline  (62%,  n  =  50)  and 
SOF  leaders  (55%,  n  =  51)  more  frequently  responded  that  the  DLAB  is  an  accurate  measure  of  language 
learning  aptitude  than  SOF  operators  (46%,  n  =  411;  Table  1,  p.  14).  This  trend  was  also  found  for  the  use 
of  DLAB  scores  for  language  assignment,  such  that  62%  ( n  =  51)  of  those  in  the  training  pipeline  and 
71  %(n  =  65)  of  SOF  leaders  indicated  that  the  DLAB  should  be  used  to  assign  individuals  to  a  language, 
compared  to  only  49%  ( n  =  431)  of  SOF  operators  (Table  2,  p.  14).  The  higher  percentage  of  leaders  and 
current  trainees  indicating  that  the  DLAB  should  be  used  for  language  assignment  may  be  explained  by 
their  relative  familiarity  with  how  the  test  scores  are  used  or  should  be  used  in  the  assignment  process  due 
to  their  current  roles  in  the  organization.  Overall,  these  findings  indicate  that  some  SOF  personnel  are 
skeptical  of  the  DLAB’s  accuracy  and  the  appropriateness  of  using  it  to  assign  individuals  to  language. 
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Further  analysis  indicated  there  were  four  groups  of  operator  respondents  (Table  3,  p.  15): 

•  DLAB  is  not  accurate  and  should  not  be  used  to  assign  individuals  to  language  training  (43%,  n  = 
377) 

•  DLAB  is  accurate  and  it  should  be  used  to  assign  individuals  to  language  training  (38%,  n  =  330) 

•  DLAB  is  not  accurate  but  it  should  be  used  to  assign  individuals  to  language  training  (10%,  n  = 
90) 

•  DLAB  is  accurate  but  should  not  be  used  to  assign  individuals  to  language  training  (8%,  n  =  73) 

These  four  groups  were  used  to  segment  and  analyze  the  open-ended  comments.  Respondents  from  the 
first  two  groups  provided  comments  referencing  their  own  experience  with  the  DLAB  and  subsequent 
language  learning  or  performance  (Table  4,  p.  16).  Comments  from  those  who  said  the  test  is  accurate  but 
should  not  be  used  for  language  assignment  suggest  that  other  situational  factors  should  be  taken  into 
account  in  the  language  assignment  process  in  combination  with  DLAB  scores,  such  as  personal 
motivation  to  learn  a  particular  language  or  previous  experience  in  learning  a  foreign  language. 

Comments  provided  by  respondents  that  said  the  DLAB  is  not  accurate  but  it  should  be  used  in  language 
assignment  reveal  the  perspective  that  the  DLAB  is  the  best  (or  only)  available  measure. 

Whereas  the  largest  group  of  operators  indicated  the  DLAB  is  not  accurate  and  should  not  be  used, 
leaders  had  a  slightly  different  view  of  the  DLAB  (Table  5,  p.  17): 

•  DLAB  is  accurate  and  should  be  used  for  language  assignment  (51%,  n  =  47) 

•  DLAB  is  not  accurate  and  should  not  be  used  for  language  assignment  (25%,  n  =  23) 

•  DLAB  is  accurate  but  should  not  be  used  for  language  assignment  (4%,  n  =  4) 

•  DLAB  is  not  accurate  but  should  be  used  for  language  assignment  (20%,  n  =  1 8) 

Overall,  leaders  more  frequently  reported  that  the  DLAB  is  accurate  and  it  should  be  used  for  language 
assignment  than  operators.  However,  their  comments  were  similar  to  operators  (Table  6,  p.  18). 

Detailed  Findings 

DLAB  accuracy 

Overall,  SOF  personnel  were  divided  on  their  perspectives  concerning  the  DLAB’s  accuracy  of 
measuring  language  learning  aptitude.  Less  than  half  of  SOF  operators  (46%,  n-  411)  indicated  that  the 
DLAB  is  an  accurate  reflection  of  their  language  learning  aptitude  (Table  1,  p.  14).  Alternatively, 
respondents  currently  in  the  training  pipeline  more  frequently  indicated  that  the  DLAB  is  an  accurate 
reflection  (62%,  n  =  50)  than  SOF  operators  ( yj=  7.20,  df=  1  ,p  <  .01).  Pipeline  respondents  may  have 
based  their  response  on  their  current  language  training  experience  and  how  their  DLAB  score  predicted 
their  current  performance  in  the  classroom.  Also,  these  trainees  may  not  have  received  an  official 
proficiency  test  yet  and  may  have  no  objective  basis  to  judge  the  accuracy  of  the  DLAB.  Furthermore, 
SOF  leaders  more  frequently  indicated  that  the  DLAB  is  an  accurate  measure  of  language  learning 
aptitude  (55%,  n  =  51)  than  operators  (x2  =  4.45,  df=  1  ,p<  .05).  Leader  comments  suggest  that  they 
based  their  responses  on  their  personal  accounts  of  DLAB  scores  and  subsequent  performance. 
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Table  1.  DLAB  Accuracy 


Group 

n 

Yes 

No 

SOF  operators 

890 

46% 

54% 

Pipeline 

81 

62% 

38% 

SOF  leaders 

92 

55%  1 

45%  I 

- - ! “ - — 7 - 

Note.  SOF  leaders  responded  ‘Yes’  significantly  more  than  SOF  operators  (x  =  4.45,  df=  1  ,p<  .05).  Pipeline  respondents 
reported  significantly  more  ‘Yes’  responses  than  SOF  operators  (%2=  7.20,  df=  1,P  <  01). 

Using  DLAB  for  language  assignment 

Similar  to  DLAB  accuracy  responses,  SOF  personnel  were  split  regarding  whether  or  not  the  DLAB 
should  be  used  to  assign  individuals  to  languages.  Slightly  less  than  half  of  SOF  operators  (49%,  n  =  431) 
indicated  that  the  DLAB  should  be  used  to  assign  individuals  to  languages  (Table  2,  p.  14).  Similarly, 
most  leader  (71%,  n  =  65)  and  training  pipeline  (62%,  n  =  51)  respondents  indicated  that  the  DLAB 
should  be  used  to  assign  individuals  to  languages.  The  higher  percentage  of  leaders  and  current  trainees 
indicating  that  the  DLAB  should  be  used  for  language  assignment  may  be  explained  by  their  relative 
familiarity  with  how  the  test  scores  are  used  or  should  be  used  in  the  assignment  process  due  to  their 
current  roles  in  the  organization.  Most  operator  responses  indicating  that  the  test  should  not  be  used  for 
language  assignment  were  expected  because  of  the  percentage  of  operators  who  indicated  that  the  DLAB 
is  not  an  accurate  measure  of  language  learning  aptitude. 

Table  2.  Using  DLAB  for  Language  Assignment 


Group 

n 

Yes 

No 

SOF  operators 

888 

49%  [ 

51% 

Pipeline 

82 

62% 

38% 

SOF  leaders 

92 

71%  1  [ 

29% 

- 2 - 

Note.  SOF  leaders  responded  ‘Yes’  significantly  more  than  SOF  operators  (%  =  9.20,  df=  1  ,p<  .05).  Pipeline  respondents 
reported  significantly  more  ‘Yes’  responses  than  SOF  operators  (%2  =  5.60,  df=  l,p  <  .05). 

Relationship  between  accuracy  and  assignment 

Thirty-eight  percent  ( n  =  330)  of  operators  indicated  that  the  DLAB  is  an  accurate  measure  of  language 
learning  aptitude  and  it  should  be  used  for  language  assignment  (Table  3,  p.  15).  These  respondents 
referenced  their  own  experience  with  DLAB  scores  and  subsequent  language  learning  or  performance 
(e.g.,  received  low  DLAB  score  and  performed  poorly  in  language  training;  received  high  DLAB  score 
and  performed  well;  Table  4,  p.  16). 

Forty-three  percent  (n  =  377)  of  operators  indicated  that  the  DLAB  is  not  an  accurate  measure  and  should 
not  be  used  for  language  assignment  (Table  3,  p.  15).  Many  of  these  respondents  referenced  their  own 
experience  with  DLAB  scores  and  subsequent  language  learning  or  performance  (e.g.,  received  low 
DLAB  score  and  performed  well;  received  high  DLAB  score  and  performed  poorly;  Table  4,  p.  16). 
Additionally,  some  respondents  reported  that  motivation  was  not  measured  by  the  DLAB  and  it  is  an 
important  component  of  one’s  ability  to  learn  a  language.  Lastly,  a  few  operators  reported  that  they  were 
not  sure  what  the  DLAB  measured. 
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A  subset  of  operators  indicated  that  the  DLAB  does  not  accurately  reflect  language  learning  aptitude  but 
it  should  be  used  to  assign  individuals  to  language  (10%,  n  =  90;  Table  3,  p.  15).  These  respondents 
commented  that  the  DLAB  is  the  best  available  measure  of  language  learning  aptitude  (Table  4,  p.  16). 

Additionally,  a  subset  of  operators  indicated  that  the  DLAB  is  an  accurate  reflection  of  language  learning 
aptitude  but  it  should  not  be  used  to  assign  individuals  to  language  (8%,  n  =  73;  Table  3).  These 
respondents  commented  that  the  language  assignment  process  does  not  take  into  account  personal  interest 
and  motivation  to  learn  a  language  (Table  4,  p.  16)  and  that  these  factors  should  be  incorporated  into  the 
language  assignment  process,  if  possible. 

Table  3.  Relationship  between  accuracy  and  language  assignment — Operator  responses 


Yes,  DLAB  accurately  reflects 
language  learning  aptitude 

No,  DLAB  does  not  accurately 
reflect  language  learning 
aptitude 

Yes,  should  be  used  to  assign 

330  (38%) 

90  (10%) 

No,  should  not  be  used  to 
assign 

73  (8%) 

377  (43%) 

Note.  Total  n  =  870;  May  not  add  to  100%  due  to  rounding  error 
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Table  4.  Exemplar  Comments  for  Each  Combination  of  DLAB  Accuracy  and  Use  for  Assignment  Perceptions — Operator  responses 


Yes,  DLAB  accurately  reflects  language  learning  aptitude 

No,  DLAB  does  not  accurately  reflect  language  learning 

aptitude 

“I  have  been  blessed  with  an  ability  to  learn  languages  since  I  was  small.  I  score  a  123  on  the 

“While  I  think  it  mis-indicated  my  personal  aptitude,  I  do  not  know  of  another 

DLAB.” 

test/method  for  better  determining  aptitude.” 

“However  well  I  can  naturally  speak  the  language  i've  learned,  compared  to  my  peers,  I  feel  that 

“What  other  tool  do  we  use?  If  we  have  another  means  then  I  would  prefer  to  drop 

our  DLAB  scores  accurately  represent  our  language  learning” 

the  DLAB” 

“The  rate  which  I  learned  a  language  was  what  I  expected  after  taking  the  DLAB.” 

“Unless  there  is  a  better  test,  it  the  only  one  for  now . ” 

“I  scored  low  which  put  me  in  a  CAT  one  language.  I  have  diffiuclty  with  the  language  so  I 
believe  the  score  was  accurate.” 

“No  other  way  to  measure  language  aptitude.” 

“Maybe  the  best  thing  out  there.” 

“My  DLAB  score  seems  to  correspond  with  how  well  I  feel  I  can  pick  up  a  language.” 

“Still  better  than  nothing  for  aptitude.” 

“My  DL  AB  score  was  very  high,  and  I  consider  myself  quick  to  understand  the  concepts 
underlying  languages.” 

“I  don't  have  a  better  way.” 

“I  scored  a  136,  and  I  am  able  to  use  5  languages  with  varying  degrees  of  proficiency.” 

“While  I  think  it  mis-indicated  my  personal  aptitude,  I  do  not  know  of  another 
test/method  for  better  determining  aptitude.” 

“Prior  experience,  the  individual's  ethnicity  and  personal  preference  (motivator)  should  be 

“I  scored  1 12  on  the  DLAB  indicating  that  I  have  a  high  aptitude  to  learn  a  foreign 

considered  when  assigning  languages.” 

language.  I  believe  that  this  overestimated  my  ability. 

“I  think  it  should  be  used  in  conjunction  with  the  language  background  of  an  individual.” 

“I  scored  a  135  on  my  DLAB  compared  to  fellow  soldiers  in  my  class  that  scored  95 
or  lower,  yet  achieved  the  same  score  (2  listening,  2  reading)  on  the  DLPT  after” 

“I  believe  interest  in  a  particular  language  is  a  factor  which  is  ignored.  Interest  is  a  factor  that  I 
think  would  increase  interest,  focus  and  other  factors  which  are  important  to  learning  proficiency.” 

“I  learned  English  after  learning  to  speak  2  other  languages.  I've  taken  the  DLAB  it 
showed  that  I  don't  have  the  ability  to  learn  a  language,  how  can  that  be  possible 

“I  think  an  individuals  interest  in  a  culture  and  language  should  dictate  his  assignment  to  a 

when  I  speak  close  to  4  languages.” 

language.  Individuals  who  are  interested  in  Chinese  culture  and  Mandarin  but  do  not  score  well 
on  the  DLAB  will  do  much  better  in  studying  Mandarin  than  they  would  studying  French,  because 

“Because  according  to  the  DLAB  I  am  unable  to  learn  a  language  when  in  fact  I  have 

they  are  interested  in  the  language  and  culture.” 

learned  two  seperate  languages  without  much  issue.” 

“In  the  end,  diligence  and  persistence  will  always  reign  over  a  single,  "cookie-cutter"  test. 

“I  scored  poorly  on  the  DLAB,  but  I  did  very  well  in  high  school  with  languages  and 

Everyone  has  a  bad  day  -  failure  happens  to  us  all  at  time.  However,  it  takes  a  conscious,  focused 

in  language  school  I  did  very  well,  according  to  the  DLAB  I  shouldn't  be  able  to 

effort  to  succeed  at  anything,  specifically  a  language.  There  are  very  few  90-minute  tests  out  there 
that  will  provide  an  accurate  teset  of  one's  will  and  determination  to  learn  (and  succeed  at)  a 
foreign  language.  The  DLAB  is  not  a  definitive  indicator.” 

learn  any  languages.” 

Note.  Comments  presented  in  this  table  are  exemplar  comments  and  do  not  represent  comments  from  the  entire  survey  sample. 
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For  leaders,  the  relationship  between  accuracy  and  assignment  ratings  were  similar  to  operators,  although 
more  leaders  reported  that  the  DLAB  is  accurate  and  should  be  used  to  assign  personnel  to  language 
training  (51%;  Table  5,  p.  17).  Leader  open-ended  comments  from  all  four  groups  were  similar  to 
operators  (Table  6,  p.  18). 

Table  5.  Relationship  between  accuracy  and  language  assignment — Leader  responses 


Yes,  DLAB  accurately  reflects 
language  learning  aptitude 

No,  DLAB  does  not  accurately 
reflect  language  learning 
aptitude 

Yes,  should  be  used  to  assign 

47  (51%) 

18  (20%) 

No,  should  not  be  used  to 
assign 

4  (4%) 

23  (25%) 

Note.  Total  n  =  92 
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Table  6.  Exemplar  Comments  for  Each  Combination  of  DLAB  Accuracy  and  Use  for  Assignment  Perceptions — Leader  responses 


Yes,  DLAB  accurately  reflects  language  learning  aptitude 

No,  DLAB  does  not  accurately  reflect  language  learning 

aptitude 

© 

QJ 

^  OJD 

“People  I  have  seen  score  well  have  an  aptitude  for  languages.” 

“It  is  an  indicator.  Better  than  nothing.” 

©  a 

“From  what  I've  seen  most  guys  who  do  well  on  the  DLAB  can  learn  languages  without  too  much 

“It's  all  we've  got  right  now.” 

3  ex 

difficulty.” 

cj  S 
,fi  c« 

“I  scored  low  on  the  DLAB  the  first  time  I  took  the  test.  The  second  time  I  barely  past.  I  failed  a 

“It  is  the  only  tool  we  currently  have  at  our  disposal” 

.3  o 

3  ~ 

language  in  High  School  and  barely  past  Language  school  in  SWC.” 

“Unless  the  soldier  can  demonstrate  his  aptitude  in  another  way,  or  already  has  an 

o  2 

JS  es 

x  3 

“Soldiers  who  do  better  on  the  DLAB  tend  to  do  better  on  the  DLPT,  but  it  is  not  a  perfect 

existing  language  he  can  expound  upon,  teh  DLAB  shows  at  least  a  bacic  measure  of 
ability.” 

PQ  .3 

<  > 

predictor.” 

hJ  TS 

“One  person  in  my  language  class  (Arabic  SOLT)  initially  had  too  low  of  a  DLAB  score  to 

O  .5 

2  a 
£ 

r\  ify 

automatically  be  in  my  class.  He  had  to  talk  with  the  class  manager  to  get  into  the  class,  and  he 
was  the  only  one  in  my  class  to  fail  the  class.  His  participation  was  more  of  a  hinderance,  because 
it  used  up  a  majority  of  the  class  time  trying  to  get  him  to  the  same  level  as  the  rest  of  the  8 -person 

C0  f/5 

Z  " 

class.” 

o 

“Though  the  DLAB  is  a  good  barometer  of  the  ability  to  learn  a  ANY  language,  some  individuals 

“I  know  many  individuals  who  have  failed  the  DLAB  but  have  gone  on  to  learn 

are  more  attuned  to  the  nuances  of  a  specific  language;  this  is  not  related  to  the  DLAB  score.  A 

categroy  II  and  even  category  III  languages.  /  /  I  have  also  noticed  that  some 

g  ox 

2  c « 

person  with  a  score  of  121  may  be  able  to  learn  Korean  quite  easily  (due  to  an  affinity  or  interest 

commands  do  not  use  the  DLAB  when  selecting  individuals  for  language  training. 

3  s 

in  the  language  structure)  yet  struggle  with  French  or  Indo  due  to  a  lack  of  interest  or  affinity  for 

The  DLAB  is  supposed  to  be  used  to  identify  which  particual  level  of  language  the 

©  OX 

-Q  © 

the  languages  nuances.” 

student  would  most  likely  be  successful  in  learning.  I  have  seen  individuals  selected 

to  learn  Arabic  when  they  scored  an  85  on  the  DLAB.  This  is  setting  the  Soldier  up 

© 

A  © 

“Aptitude  and  the  ability  are  completely  different  things.” 

for  failure  and  is  wasting  money  and  time  that  could  be  better  spent  training  the 

2  2 

soldier  an  easier,  but  just  as  important,  language.” 

S 

®  3 

“Antiquated  test.  I  hold  a  proficiency  rating  of  3/3  in  two  languages  and  did  not 

PQ  It 

score  exceptionally  high  on  the  DLAB.” 

<  1 

“Students  with  low  DLAB  scores  have  been  enrolled  and  completed  higher  category 

a  h 

© 

languages.” 

^  *f/3 
^  t/3 

“Native  speakers  do  not  score  well  on  the  DLAB,  yet  speak  the  native  language 

© 

z 

better  than  English.” 

Note.  Some  comments  presented  in  this  table  are  exemplar  comments  and  do  not  represent  comments  from  the  entire  survey  sample  (i.e.,  comments  in  the  Yes/Yes  and  No/No 
quadrants).  Comments  in  the  No/Yes  and  Yes/No  quadrants  include  all  comments  provided. 
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SECTION  IV:  COMMENTS  ABOUT  THE  DLAB 

Past  research  identified  procedural  justice  factors  that  contribute  to  perceptions  of  a  test’s  accuracy  and 
fairness  (Gilliand,  1993).  These  factors  include  information  known  about  the  test,  chance  to  perform, 
treatment  at  the  testing  site,  and  consistency  of  test  administration.  Information  about  the  test  refers  to 
explanation  and  communication  about  the  test  and  how  scores  will  be  used  prior  to  testing.  Chance  to 
perform  refers  to  having  the  chance  to  demonstrate  relevant  knowledge,  skills,  and/or  abilities  on  the  test. 
Treatment  at  the  test  site  refers  to  the  degree  to  which  test  takers  are  treated  with  respect.  Consistency  of 
test  administration  refers  to  test  administration  and  decision  procedure  consistency.  These  factors 
contribute  to  fairness  and  credibility  perceptions  of  the  test  (Bauer  et  al.,  1998)  and  are  gathered  by 
assessing  test  taker  reactions  to  the  test.  Therefore,  it  is  important  to  assess  SOF  personnel’s  reactions  to 
determine  if  procedural  justice  factors  influence  the  DLAB’s  credibility. 

SOF  operators  and  leaders  provided  comments  about  the  DLAB’s  accuracy,  its  use  for  assignment,  and 
other  DLAB  topics.  Focus  group  discussions  also  provided  perceptions  of  the  DLAB  and  its  use  in  the 
language  assignment  process.  Examining  these  comments  provides  insight  into  whether  test  takers  and 
others  perceive  the  DLAB  to  be  a  credible  measure  of  language  learning  aptitude. 

Information  about  response  rates  for  each  open-ended  survey  item  presented  in  this  report  is  available  in 
the  Methodology  section  (Appendix  B).  Information  about  the  focus  group  discussions  presented  in  this 
report  is  available  in  the  Participation  Report  (Technical  Report  #2010011003)  and  the  Methodology’ 
Report  (Technical  Report  #2010011002). 

Research  Questions 

This  section  addresses  the  following  question: 

•  What  factors  influence  operators’  and  leaders’  perceptions  of  the  DLAB’s  accuracy  and  its  use  in 
the  language  assignment  process? 

Main  Findings 

Operator  and  leader  comments  and  focus  group  discussions  indicated  that  the  DLAB’s  accuracy  may  be 
negatively  affected  by  multiple  factors,  including  (Table  7,  pp.  22-23): 

•  No  knowledge  of  what  the  DLAB  measures  or  how  it  is  used  in  the  language  assignment  process 
prior  to  testing 

•  Perceptions  that  the  DLAB  does  not  measure  all  aspects  of  language  learning  aptitude 

•  Perceptions  that  the  DLAB  measures  skills  unrelated  to  language  learning  (i.e.,  test- taking  skills) 

•  Perceptions  or  experiences  that  suggest  DLAB  scores  are  not  used  to  assign  individuals  to 
language 

•  Negative  testing  experiences  (e.g.,  testing  conditions,  administrative  mistakes) 
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SOF  personnel  should  be  educated  about  the  DLAB  and  what  it  measures  (i.e.,  potential  to  learn  a  foreign 
language)  and  how  it  is  used  in  the  language  assignment  process  prior  to  taking  the  test.  As  described  by 
Bauer  and  colleagues  (1998,  2001),  information  known  about  the  test  contributes  to  an  individual’s 
perception  of  whether  or  not  it  is  credible  and  fair.  Several  comments  revealed  that  some  operators  did  not 
know  what  the  DLAB  was  supposed  to  measure.  This  is  problematic  because  test  takers  cannot  make 
accurate  assessments  of  the  test’s  credibility  without  knowing  what  the  test  is  supposed  to  measure. 
Furthermore,  without  prior  knowledge  about  what  the  DLAB  measures,  motivation  to  take  the  test  and 
testing  process  seriously  may  be  negatively  influenced.  Flowever,  this  must  be  balanced  with  the 
possibility  that  some  test  takers  will  intentionally  underperform  on  the  test  as  a  strategy  to  avoid  being 
assigned  to  a  more  difficult  language.  One  alternative  strategy  is  to  inform  test  takers  of  the  use  and 
purpose  of  the  DLAB  after  they  complete  the  test  (i.e.,  debriefing).  This  debriefing  strategy  might 
improve  perceptions  of  the  test  but  does  not  address  the  pre-DLAB  motivation  issue. 

SOF  personneFs  perceptions  about  whether  the  DLAB  measures  all  aspects  of  language  learning  aptitude 
and  whether  it  measures  abilities  unrelated  to  language  learning  should  be  monitored.  As  indicated  by 
Bauer  and  colleagues  (2001),  having  a  chance  to  demonstrate  knowledge,  skills,  and  abilities  on  the  test 
contributes  to  perceptions  of  its  credibility  and  fairness.  If  the  test  does  not  measure  all  aspects  of  what  is 
supposed  to  be  measured  or  if  the  test  measures  something  unrelated,  then  the  test  may  be  perceived 
negatively.  Some  operators  pointed  out  that  the  DLAB  does  not  take  into  account  other  factors  that  could 
influence  one’s  ability  to  learn  a  language,  such  as  motivation  and  personal  interest  in  a  particular 
language.  Furthermore,  some  comments  suggested  that  the  DLAB  measures  an  individual’s  test-taking 
ability,  which  is  not  relevant  to  one’s  ability  to  learn  a  language.  These  findings  suggest  that  some  test 
takers  do  not  perceive  the  DLAB  as  credible  or  fair  because  they  do  not  think  that  it  measures  what  it  is 
supposed  to  measure.  The  DLAB  2  development  is  aimed  at  overcoming  the  deficiencies  of  the  DLAB. 

To  preserve  or  increase  perceived  test  fairness  across  the  SOF  community,  the  DLAB  should  be  used 
consistently  to  assign  individuals  to  language  training.  As  indicated  by  Bauer  and  colleagues  (2001),  test 
administration  and  decision  procedure  consistency  influences  test  perceptions.  Comments  revealed  that 
some  SOF  personnel  do  not  think  that  DLAB  scores  are  used  for  assignment  into  language  training.  Many 
of  these  comments  referenced  a  personal  experience  where  an  individual  who  scored  low  on  the  DLAB 
was  placed  into  a  difficult  (i.e.,  Category  III  or  IV)  language  class.  If  other  factors  (e.g.,  current  and  future 
force  requirements,  previous  language  experience)  are  incorporated  into  the  assignment  decision  or  can 
override  the  DLAB  in  the  assignment  decision,  the  full  assignment  criteria  and  process  should  be 
communicated  to  personnel  at  the  appropriate  time  in  the  process. 

Finally,  SOF  personnel  should  be  provided  with  reasonable  testing  accommodations.  Conditions  and 
treatment  at  the  testing  site  can  influence  whether  test  takers  perceive  the  test  as  credible  and  fair  (Bauer 
et  al.,  2001).  For  example,  operators  from  USASOC  reported  that  the  temperature  in  their  testing 
environment  was  too  hot  and  that  they  did  not  have  an  adequate  amount  of  sleep  the  night  before  the  test 
due  to  other  requirements.  These  factors  can  systematically  impact  individual  test  performance  for  those 
who  experienced  these  testing  environment  constraints  and,  therefore,  alter  or  undermine  the  predictive 
validity  of  the  test,  which  can  lead  to  inaccurate  assignment  and  diminished  perceptions  of  credibility  and 
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fairness.  If  these  administrative  constraints  are  expected  to  occur  in  the  future,  research  should  be 
conducted  under  these  constrained  conditions  to  determine  the  impact  on  the  predictive  validity  of  the 
DLAB.  Furthermore,  other  test  administration  issues  were  reported  (e.g.,  test  administrator  not  taking  the 
test  seriously  or  not  providing  clear  instructions). 

Detailed  Findings 

Operators  and  leaders  provided  comments  relating  to  the  DLAB’s  accuracy,  its  use  in  language 
assignment,  and  other  specific  feedback  about  the  test  (Table  7,  p.  22-23).  Overall,  respondents 
commented  on  many  factors  that  can  influence  an  individual’s  perception  of  the  DLAB  as  a  fair  and 
credible  test.  These  factors,  which  were  identified  by  analysis  of  the  survey  comments  and  focus  group 
discussions,  included: 

•  No  knowledge  of  what  the  DLAB  measures  or  how  it  is  used  in  the  language  assignment  process 
prior  to  taking  it 

•  Perceptions  that  the  DLAB  does  not  measure  all  aspects  of  language  learning  aptitude 

•  Perceptions  that  the  DLAB  measures  skills  unrelated  to  language  learning  (i.e.,  test- taking  skills) 

•  Perceptions  of  experiences  that  suggest  DLAB  scores  are  not  used  to  assign  individuals  to 
language 

•  Negative  testing  experiences  (e.g.,  testing  conditions,  administrative  mistakes) 

These  factors  align  with  factors  identified  by  past  research  (Bauer  et  al.,  1998,  2001;  Gilliand,  1993). 
These  factors,  gathered  from  test  taker  reactions,  include  information  known  about  the  test,  chance  to 
perform,  and  treatment  at  the  testing  site.  These  reactions  contribute  to  test  accuracy  and  fairness 
perceptions  (Bauer  et  al.,  1998). 
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Table  7.  Comment  Theme  Frequencies 


Overall 

Operator 

Leader 

DLAB  is  accurate 

General  comment  that  DLAB  accurately  measures  language  learning  aptitude 

82 

64 

18 

Received  high  DLAB  scores  and  performed  well  in  language 

20 

15 

5 

General  positive  DLAB  comment 

12 

7 

5 

Received  low  DLAB  score  and  performed  poorly  in  language 

12 

11 

1 

DLAB  is  inaccurate 

DLAB  doesn’t  account  for  motivation/attitude/effort 

93 

75 

18 

General  comment  that  DLAB  inaccurately  measures/fails  to  measure  language  learning  aptitude 

58 

52 

6 

Received  low  DLAB  score  and  does  well  in  difficult  language 

67 

61 

8 

General  negative  DLAB  comment 

44 

38 

6 

Other  factors  that  DLAB  doesn’t  account  for  (e.g.,  study  habits,  learning  strategies) 

43 

34 

9 

DLAB  not  based  on  a  real  language 

24 

22 

2 

Received  high  DLAB  score  and  does  poorly  in  difficult  languages 

15 

14 

1 

Some  people  aren’t  good  at  taking  tests 

10 

9 

1 

Testing  condition  issues  with  DLAB 

10 

10 

0 

DLAB  doesn’t  test  all  aspects  of  language  learning  aptitude  (e.g.,  does  not  consider  grammar,  conjugations,  etc) 

5 

5 

0 

DLAB  was  more  difficult  than  learning  the  language 

2 

1 

1 

Note.  Some  comments  contained  multiple  themes.  Therefore,  the  total  number  of  codes  assigned  may  be  greater  than  the  total  number  of  comments. 
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Table  7  (continued).  Comment  Theme  Frequencies 


Overall 

Operator 

Leader 

Other 

Suggestions  regarding  language  assignments 

94 

76 

18 

Not  relevant  to  the  DLAB 

80 

76 

4 

Perceptions  of  what  the  DLAB  tests 

48 

38 

10 

DLAB  is  the  best/only  available  language  aptitude  test 

26 

17 

9 

Lack  of  knowledge  about  DLAB 

22 

21 

1 

DLAB  scores  aren’t  used  to  assign  people  to  languages 

14 

11 

3 

Note.  Some  comments  contained  multiple  themes.  Therefore,  the  total  number  of  codes  assigned  may  be  greater  than  the  total  number  of  comments. 
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As  described  by  Bauer  and  colleagues  (1998,  2001),  information  known  about  the  test  contributes  to  an 
individual’s  perception  of  whether  or  not  the  test  is  credible  and  fair.  A  test’s  credibility  could  be 
diminished  by  test  takers’  perceptions  that  the  test  measures  something  other  than  what  it  claims  to 
measure,  or  that  the  test  does  not  measure  all  components  of  what  it  claims  to  measure.  An  issue  arises 
when  test  takers  do  not  know  what  the  test  is  meant  to  measure.  Test  takers  must  know  what  the  test  is 
measuring  to  help  them  evaluate  its  credibility  and  fairness.  Otherwise,  they  will  probably  discredit  the 
test.  Some  operators  reported  that  they  did  not  have  knowledge  of  what  the  DLAB  measured  before  they 
took  the  test.  Exemplar  verbatim  comments  are  provided  below. 

“i  have  no  idea  how  it  is  scored  that  was  never  explained  in  any  way  to  me” 

SOF  Operator,  USSOCOM  Operational  unit 

“I  do  not  understand  the  scoring  of  the  DLAB  ” 

SOF  Operator,  4th  POG 

“7  had  no  idea  what  this  test  is  supposed  to  reflect.  ” 

SOF  Operator,  “Other”  Operational  unit 

“I  have  no  relevent  opinion  on  this  question.  I  have  no  idea  how  they  use  the  results  to 
assign  personnel.  ” 

SOF  Operator,  10th  SFG 

As  indicated  by  Bauer  and  colleagues  (2001),  having  a  chance  to  demonstrate  knowledge,  skills,  and 
abilities  on  the  test  contributes  to  perceptions  of  its  accuracy  and  fairness.  If  test  takers  think  the  test  does 
not  measure  the  entire  scope  of  what  is  supposed  to  be  measured,  it  may  decrease  the  test’s  perceived 
credibility.  Many  operators  and  leaders  said  that  the  DFAB  does  not  account  for  other  factors  that  could 
influence  an  individual’s  language  learning  aptitude,  such  as  motivation  or  personal  interest  in  the 
language. 

“Learning  a  foreign  language  really  depends  on  the  Soldiers  motivation  to  learn  the 
language.  A  Soldier  can  get  a  high  score  on  the  DLAB  but  have  no  interest  in  the 
language  they  are  given  to  learn.  If  a  Soldeir  has  no  motivation  to  learn  a  language  they 
are  not  going  to  retain  it.  ” 

SOF  Operator,  4th  POG 

“It  does  not  account  for  work  ethic.  During  language  training  I  witnessed  many  students, 
who  scored  well  on  their  DLAB,  put  forth  little  to  no  effort  in  leaning  their  language  ” 

SOF  Operator,  4th  POG 

“DLAB  scores  do  not  accurately  reflect  language  apitutde  because  motivation  to  learn  is 
not  measured.  ” 

SOF  Operator,  95th  CAB 

“7  know  of  individuals  who  have  had  low  DLAB  scores,  but  do  very  well  in  foreign 
language,  because  of previous  experience  in  that  foreign  language.  I  also  know  of  other 
individuals  who  have  scored  well  on  the  DLAB,  but  have  performed  poorly  in  the 
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assigned  foreign  language,  because  the  language  was  assigned  and  there  was  no 
personal  motivation.  ” 

SOF  Operator,  DLI 

Just  as  credibility  can  be  negatively  affected  by  the  test  not  measuring  all  the  components  of  what  it  is 
supposed  to  be  measuring,  it  can  also  be  adversely  affected  by  including  irrelevant  components.  Some 
respondents  said  that  performance  on  the  DLAB  is  contingent  upon  having  strong  test-taking  skills,  which 
are  unrelated  to  an  individual’s  ability  to  learn  a  language. 

“It  did  not  make  any  sense.  Some  people  just  don't  test  well.  That  doesn't  mean  they  can 
not  learn.  ” 

SOF  Operator,  4th  POG 

“Again  this  measure  you  ability  to  take  a  test.  College  grads  do  well  and  high  school 
grads  are  lacking.  The  high  school  grads  simply  don't  have  the  experience  in  takin  test.  ” 

SOF  Operator,  7th  SFG 

“ Some  people  simply  do  not  take  tests  well.  Their  results  do  not  accurately  reflect  their 
aptitude.  ” 

SOF  Operator,  USSOCOM 

“I  think  that  it  is  a  matter  of  who  is  good  on  a  multiple  choice  test” 

SOF  Operator,  95th  CAB 

Participant:  “Are  you  guys  familiar  with  the  DLAB?” 

Moderator:  “The  DLAB,  yes.  ” 

Participant:  “It ’s  a  made-up  language.  ” 

Moderator:  “Yes.  ” 

Participant:  “I  don 't  see  why — I  mean,  some  people  just  don 't  test  well.  I  mean,  honestly, 
some  people  just  don 't  test  well.  They  're  really  smart  or  something,  but  if  you  give  it  to 
them  written  in  a  test,  they’re  like  ‘Huh. 

WARCOM  focus  group 

As  described  by  Bauer  and  colleagues  (2001),  test  administration  and  decision  procedure  consistency 
influences  test  perceptions.  If  test  scores  are  not  being  used  properly,  test  motivation  may  be  negatively 
affected.  Some  operators  reported  that  DLAB  scores  are  not  used  as  the  policy  stipulates  to  assign 
individuals  to  languages  for  training. 

“It  should  be  used  but  I  got  like  a  73  and  they  gave  me  arabic  so  they  don 't  use  the  DLAB 
for  anything.  ” 

SOF  Operator,  5th  SFG 

“I  think  this  is  fairly  obvious.  Don 't  assign  Chinese  or  Russian  to  someone  who  barely 
passed  the  DLAB  ...  it's  done  all  the  time.  I  don't  understand  why.  ” 

SOF  Operator,  USSOCOM  operational  unit 
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“I  did  not  score  high,  and  still  got  a  cat4  language.  The  test  was  worthless  ” 

SOF  Operator,  5th  SFG 

“I  do  not  believe  that  the  DLAB  is  used  to  assign  languages.  I  think  that  it  has  been  more 
a  needs  of  the  Groups  and  what  languages  are  native  to  those  Groups  that  has  been 
driving  the  languages  that  students  learned  while  in  training.  This  is  speaking  for  18 
series  guys  only.  ” 

SOF  Operator,  Deployed  SO  unit 

“I  have  noticed  that,  on  average,  those  with  a  higher  DLAB  score  tend  to  do  better  than 
those  with  a  lower  score.  Of  course  nothing  is  a  better  indicator  than  genuine 
enthusiasm  to  learn  another  language,  and  those  Soldiers  need  to  be  identified,  assessed, 
and  trained  accordingly.  ” 

SOF  Operator,  5th  SFG 


‘Different  learning  styles  may  prevent  students  from  being  as  successful  during  DLAB  as 
they  are  during  DLPT  based  on  self  study  and  motivation  during  the  course  work.  ” 

SOF  Operator,  USAREC 

“it  doesn 't  gauge  that  some  people  learn  language  in  different  ways,  and  that  also  a 
persons  interpersonal  skills  can  make  them  much  more  effective,  basically  if  your  well 
liked  the  locals  will  also  try  harder  to  understand  you  and  work  with  you  ” 

SOF  Operator,  MARSOC 

Conditions  and  treatment  at  the  testing  site  can  influence  whether  test  takers  perceive  the  test  as  credible 
and  fair  (Bauer  et  al.,  2001).Test  takers  need  adequate  testing  conditions  so  they  can  take  the  test  to  the 
best  of  their  ability.  Some  operators,  all  of  which  were  from  USASOC,  reported  that  the  temperature  in 
their  testing  room  was  too  hot.  Also,  some  operators  reported  lacking  sleep  the  night  before  the  test  due  to 
other  requirements.  Other  testing  administrative  issues  were  also  reported,  such  as  the  test  administrator 
not  taking  the  test  seriously  or  not  providing  clear  instructions. 

“I fell  asleep  when  I  took  the  DLAB  because  or  inadquate  facilities  (the  room  was  100 
degrees  in  the  middle  of  a  hot  North  Carolina  summer)  and  because  our  drill  sergeants 
did  not  let  us  get  enough  sleep  the  night  before  and  the  took  us  on  a  8  mile  run  right 
before  taking  the  test.  The  whole  situation  set  everyone  who  took  that  test  up  for  failure.  ” 

SOF  Operator,  4th  POG 


“I  was  tired  and  stressed  when  I  took  it  in  SFAS.  ” 

SOF  Operator,  4th  POG 

“The  test  was  given  under  stressful  condictions  ” 

SOF  Operator,  7th  SFG 
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“During  the  course  of  my  DLAB  I  was  unaware  of  a  final  section  of  the  exam  until  very > 
little  time  remained.  As  a  result  I  was  unable  to  adequately  attend  to  the  questions  in  this 
section  and  my  score  consequently  suffered.  ” 

SOF  Operator,  10th  SFG 

“I  took  it  when  I  was  a  brand  new  LT;  I  didn 't  know  what  it  was,  and  when  the  'proctor' 
said,  7  don 't  care  if  you  finish  it, '  we  all  left.  ” 

SOF  Operator,  ARGENT 

“Many  soldiers  dont pay  attention  to  the  'jargon  mumbo  jumbo'  in  the  test  after  it  has 
began  ” 

SOF  Operator,  10th  SFG 

“As  far  as  the  testing  in  school  it ’s  really  not  an  accurate  way  to  test  somebody ’s  ability 
to  learn  a  foreign  language  given  the  situation,  the  way  the  testing  is  done,  the  focus  to 
the  student  should  be  testing  after  that  school  situation  where  they  're  not  training  all  day 
and  coming  to  this  test,  haven 't  had  time  to  go  to  sleep.  ” 

WARCOM  focus  group 
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SECTION  V:  CONCLUSIONS  AND  RECOMMENDATIONS 

Assigning  individuals  to  an  appropriate  language  for  training  is  an  important  decision  with  many  potential 
implications.  If  individuals  are  not  accurately  assigned,  there  can  be  financial  and  capability 
consequences  for  the  organization,  performance  consequences  for  the  unit  and  individual,  and 
motivational  consequences  for  the  individual.  If  a  person  is  not  properly  assigned,  it  is  an  ineffective  use 
of  time  and  money  for  the  organization.  For  example,  in  the  Special  Forces  Qualification  Course  (SFQC), 
if  an  individual  is  assigned  to  train  in  a  language  that  is  too  difficult  and  fails  to  meet  the  proficiency 
standard,  then  additional  financial  and  time  investments  are  needed  to  raise  that  person’s  proficiency  to  an 
acceptable  level  for  graduation  or  the  total  investment  in  that  person’s  selection  and  training  is  lost  (i.e., 
he  does  not  finish  the  SFQC  and  does  not  enter  SF).  Furthermore,  if  assignment  is  ineffective,  individuals 
may  arrive  at  their  unit  without  the  proper  language  skills  or  with  insufficient  levels  of  skill  to  support 
their  unit’s  mission.  If  individuals  are  placed  into  the  wrong  language  or  fail  to  achieve  or  maintain 
proficiency,  then  their  skills  and  knowledge  cannot  be  fully  utilized  in  their  unit.  Again,  ineffective 
language  assignment  has  financial,  capability,  and  mission  performance  consequences  for  SOF,  the  unit, 
and  the  individual.  For  example,  if  adequate  language  skills  are  not  available,  then  the  unit  must  utilize 
interpreters,  which  have  financial  and  mission  effectiveness  consequences.  Finally,  trainees  may  become 
discouraged  if  placed  in  a  language  that  is  too  difficult,  or  bored  if  placed  in  a  language  that  is  too  easy. 
This  may  affect  their  motivation  to  learn  the  language,  maintain  the  language,  use  the  language  on 
missions,  and  continue  learning  the  language  once  at  their  unit. 

The  DLAB  is  currently  used  for  language  assignment  in  the  SOF  community.  Although  research  shows 
the  DLAB  is  a  limited  predictor  of  success  in  language  training  (e.g.,  SWA  Consulting,  Technical  Report 
#2009010616,  Technical  Report  #2009010612),  it  is  the  best  single  measure  of  language  learning  aptitude 
currently  available.  Until  a  better  alternative  is  developed,  the  value  of  the  DLAB  must  be  optimized 
through  effective  use.  Consistent  and  effective  use  will  maximize  the  validity  of  the  DLAB  for  language 
assignment  and  result  in  optimized  outcomes  for  SOF,  units,  and  individuals. 

In  addition  to  the  statistical  properties  of  the  DLAB  (i.e.,  reliability  and  validity),  the  DLAB  needs  to  be 
perceived  as  an  accurate  measure  by  SOF  operators  and  leaders  so  that  its  use  for  language  assignment  is 
perceived  as  fair.  However,  this  study’s  findings  show  that  some  operators  and,  to  a  lesser  extent,  leaders, 
perceive  the  test  as  inaccurate.  Reasons  for  their  views  include: 

•  No  knowledge  of  what  DLAB  measures  or  how  it  is  used  in  the  language  assignment  process 
prior  to  taking  the  test 

•  Perceptions  that  the  DLAB  does  not  measure  all  aspects  of  learning  language  aptitude 

•  Perceptions  that  the  DLAB  measures  skills  unrelated  to  language  learning  (i.e.,  test- taking  skills) 

•  Perceptions  or  experiences  that  DLAB  scores  are  not  used  to  assign  individuals  to  language 

•  Negative  testing  experiences  (e.g.,  testing  conditions,  administrative  issues) 

These  factors  align  with  factors  identified  by  past  research  that  influence  test  accuracy  and  fairness 
perceptions  (Bauer  et  al.,  2001,  1998;  Gilliand,  1993).  To  change  operator  and  leader  perceptions  of  the 
DLAB,  each  of  these  factors  needs  to  be  addressed.  Some  of  these  factors  are  characteristics  of  the  test 
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and  can  only  be  addressed  by  improving  the  DLAB;  however,  others  can  be  addressed  through 
communication  and  effective  and  consistent  administration  and  use  of  the  DLAB.  Leaders  in  the  SOF 
community  can  advocate  for  changes  to  the  DLAB  and/or  investigate  alternatives,  but  the  SOF  leaders 
who  control  the  administration  of  the  DLAB  and  the  use  of  the  test  for  language  assignment  can  achieve 
the  most  direct  and  immediate  impact. 

First,  personnel  who  take  the  DLAB  need  explicit  explanation  as  to  what  the  DLAB  measures  and  how  it 
is  used  in  the  assignment  process  prior  to  taking  the  test,  especially  if  this  can  be  done  in  such  a  way  as  to 
not  compromise  the  DLAB’s  functioning.  As  described  by  Bauer  and  colleagues  (1998,  2001), 
information  known  about  the  test  contributes  to  an  individual’s  perception  of  whether  or  not  the  test  is 
accurate  and/or  fair.  Survey  comments  indicated  that  some  operators  did  not  know  what  the  test  was 
supposed  to  measure  prior  to  testing.  This  lack  of  knowledge  could  lead  to  negative  perceptions  about  the 
DLAB,  and  these  perceptions  may  influence  test  takers’  motivation  to  perform  well  on  the  test.  It  should 
be  noted  that  explaining  to  test  takers  what  the  DLAB  intends  to  measure  and  how  it  will  be  used  in  the 
language  assignment  process  can  lead  to  some  test  takers  intentionally  underperforming  on  the  test  as  a 
strategy  to  avoid  being  assigned  to  a  more  difficult  language.  In  this  case,  presenting  the  DLAB  as  one 
factor  in  language  assignment  process  might  be  most  beneficial.  One  alternative  strategy  is  to  inform  test 
takers  of  the  purpose  and  use  of  the  DLAB  after  they  complete  the  test  (i.e.,  debriefing).  The  debriefing 
strategy  might  improve  perceptions  of  the  test  but  does  not  address  the  pre-DLAB  motivation  issue.  Also, 
the  DLAB  is  often  administered  prior  to  individuals  entering  the  SOF  trainee  pipeline  or  the  SOF 
community.  In  this  case,  it  is  impossible  to  communicate  the  use  of  the  DLAB  in  SOF  language 
assignment  or  the  importance  of  the  test.  However,  this  communication  is  possible  when  the  DLAB  is 
administered  as  part  of  the  process  in  a  SOF  organization.  Regardless,  test  takers  need  some  information 
prior  to  testing  to  inform  their  motivation  and  engagement  in  the  DLAB  testing  process. 

Second,  perceptions  about  whether  the  DLAB  measures  all  aspects  of  language  learning  aptitude  and 
whether  it  measures  abilities  unrelated  to  language  learning  should  be  monitored.  As  indicated  by  Bauer 
and  colleagues  (2001),  having  a  chance  to  demonstrate  knowledge,  skills,  and  abilities  on  the  test 
contributes  to  perceptions  of  its  accuracy  and  fairness.  If  it  is  perceived  that  a  test  does  not  measure  all 
aspects  of  what  it  is  supposed  to  measure  or  if  the  test  measures  something  unrelated  to  what  it  should  be 
testing,  then  the  test  may  be  perceived  negatively.  For  example,  some  operators  reported  that  the  DLAB 
does  not  measure  all  aspects  of  language  learning  aptitude,  such  as  motivation  or  personal  interest  in  a 
language.  Furthermore,  some  operators  suggested  that  the  DLAB  measures  an  individual’s  test  taking 
skills,  which  are  unrelated  to  their  ability  to  learn  a  foreign  language.  These  negative  perceptions  can  be 
passed  to  individuals  who  have  not  yet  taken  the  test,  negatively  influencing  their  motivation  to  engage  in 
the  DLAB  testing  process  and  their  resulting  scores.  The  less  accurate  scores  resulting  from  unmotivated 
test  performance  may  undermine  the  statistical  properties  of  the  scores  and  the  resulting  language 
assignments.  Although  the  DLAB  and  DLAB  2  testing  programs  are  controlled  outside  of  USSOCOM, 
USSOCOM  should  communicate  its  testing  requirements  and  feedback  to  the  appropriate  groups.  An 
effective  language  aptitude  test  should  not  only  have  sufficient  psychometric  properties  (validity  and 
reliability)  and  predictive  validity  for  SOF  language  training  outcomes,  but  it  should  also  include  relevant 
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factors  and  exclude  irrelevant  factors.  Users  should  perceive  the  test  as  being  an  accurate  predictor  to 
maximize  the  effectiveness  of  its  use. 

Third,  to  preserve  or  increase  perceived  test  credibility  and  fairness,  DLAB  scores  should  be  consistently 
used  for  language  assignment  purposes.  As  indicated  by  Bauer  and  colleagues  (2001),  test  administration 
and  decision  procedure  consistency  influences  test  perceptions.  Some  respondents  indicated  that  the 
DLAB  should  not  be  used  for  language  assignment  because  of  credibility  issues.  If  factors  that  negatively 
influence  DLAB  accuracy  perceptions  can  be  addressed  then  the  fairness  perceptions  of  the  language 
assignment  process  should  improve.  A  language  assignment  process  that  is  effective  and  applied 
consistently  will  be  perceived  as  accurate  and  fair.  Although  current  and  future  force  requirements  may 
require  that  language  assignments  not  be  based  solely  on  DLAB  scores,  consistent  use  of  the  DLAB  and 
communication  of  its  use  should  improve  perceptions.  SOF  leaders  have  direct  control  over  how  the 
DLAB  is  used  in  the  assignment  process — regardless  if  it  is  the  only  factor  or  one  of  many  used — and 
they  should  ensure  the  effective  and  consistent  use  of  the  DLAB  or  its  alternative.  If  other  factors  (e.g., 
previous  language  experience,  current  and  future  force  requirements)  are  considered  in  the  language 
assignment  process  or  can  override  the  DLAB  in  the  assignment  decision,  the  full  assignment  criteria  and 
process  should  be  communicated  to  personnel  at  the  appropriate  time  in  the  process. 

Finally,  test  takers  should  be  provided  with  reasonable  testing  accommodations.  Conditions  and  treatment 
at  the  testing  site  can  influence  whether  test  takers  perceive  the  test  as  credible  and  fair  (Bauer  et  al., 
2001).  Specifically,  some  operators  reported  that  the  test  room  temperature  was  too  hot.  An  easy  fix 
would  be  to  provide  fans  or  an  air  conditioning  unit  for  the  testing  rooms.  Furthermore,  some  operators 
reported  that  they  did  not  receive  adequate  sleep  the  night  before  the  test  due  to  other  training 
requirements.  Those  responsible  for  the  test  takers  should  be  aware  of  the  testing  date  and  provide 
operators  the  opportunity  to  rest  the  night  before  the  test.  These  factors  can  systematically  influence 
individual  test  performance  in  groups  that  experienced  the  constraints  and,  therefore,  alter  or  undermine 
predictive  validity.  If  groups  differ  on  conditions,  it  could  create  two  predictive  profiles,  obscuring  the 
interpretation  of  the  test’s  scores  and  the  effectiveness  of  cut  scores  used  for  assignment  purposes.  If  these 
testing  administration  conditions  are  expected  to  occur  in  the  future,  research  should  be  conducted  under 
these  constrained  conditions  to  determine  the  impact  on  the  predictive  validity  of  the  DLAB.  Lastly,  some 
operators  reported  test  administration  issues,  such  as  the  test  administrator  not  taking  the  test  seriously  or 
not  providing  clear  test  instructions.  All  test  administrators  should  be  trained  to  provide  adequate  testing 
instructions  for  test  takers  and  maintain  a  neutral  attitude  about  the  test. 

Although  perceptions  about  the  DLAB’s  accuracy  and  its  use  for  language  assignment  were  mixed,  steps 
can  be  taken  to  improve  these  perceptions.  Two  key  actions  are  to  communicate  the  DLAB  testing  policy 
and  to  encourage  effective  and  consistent  use  of  the  DLAB  for  language  assignment  within  a  SOF 
organization.  Based  on  research  in  other  areas,  if  a  test  is  effectively  and  consistently  used  in  an 
organization,  then  perceptions  of  its  accuracy  and  fairness  will  improve.  If  perceptions  improve,  then  test 
takers  will  engage  in  the  testing  process  and  test  results  will  be  more  accurate  and  representative  of 
individual  aptitudes.  If  test  results  are  more  accurate,  then  the  test’s  use  for  assignment  will  be  more 
effective.  In  this  case,  if  perceptions  of  the  DLAB  improve,  test  takers  engage  in  the  DLAB  testing 
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process  more  seriously  and  the  DLAB’s  predictive  validity  is  optimized,  then  the  language  assignment 
will  be  more  appropriate  and  effective,  providing  the  organization,  the  unit,  and  the  individual  with  the 
best  opportunity  to  succeed  in  the  language  domain.  SOF  leaders  should  also  continue  to  identify 
alternatives  or  supplements  to  the  DLAB.  When  the  DLAB  2  is  available,  it  should  be  validated  for  use 
with  SOF  personnel,  if  the  test  is  feasible  for  administration  within  the  constraints  of  SOF  operational 
use.  Regardless  of  the  DLAB’s  psychometric  properties  or  specific  use,  it  is  currently  the  single  most 
consistent  test  for  language  assignment,  and  SOF  leaders  should  develop  use  policies  that  optimize  its 
effectiveness. 
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ABOUT  SWA  CONSULTING  INC. 

SWA  Consulting  Inc.  (formerly  Surface,  Ward,  and  Associates)  provides  analytics  and  evidence-based 
solutions  for  clients  using  the  principles  and  methods  of  industrial/organizational  (I/O)  psychology.  Since 
1997,  SWA  has  advised  and  assisted  corporate,  non-profit  and  governmental  clients  on: 

•  Training  and  development 

•  Performance  measurement  and  management 

•  Organizational  effectiveness 

•  Test  development  and  validation 

•  Program/training  evaluation 

•  Work/job  analysis 

•  Needs  assessment 

•  Selection  system  design 

•  Study  and  analysis  related  to  human  capital  issues 

•  Metric  development  and  data  collection 

•  Advanced  data  analysis 

One  specific  practice  area  is  analytics,  research,  and  consulting  on  foreign  language  and  culture  in  work 
contexts.  In  this  area,  SWA  has  conducted  numerous  projects,  including  language  assessment  validation 
and  psychometric  research;  evaluations  of  language  training,  training  tools,  and  job  aids;  language  and 
culture  focused  needs  assessments  and  job  analysis;  and  advanced  analysis  of  language  research  data. 

Based  in  Raleigh,  NC,  and  led  by  Drs.  Eric  A.  Surface  and  Stephen  J.  Ward,  SWA  now  employs  close  to 
twenty  I/O  professionals  at  the  masters  and  PhD  levels.  SWA  professionals  are  committed  to  providing 
clients  the  best  data  and  analysis  upon  which  to  make  evidence-based  decisions.  Taking  a  scientist- 
practitioner  perspective,  SWA  professionals  conduct  model-based,  evidence-driven  research  and 
consulting  to  provide  the  best  answers  and  solutions  to  enhance  our  clients’  mission  and  business 
objectives.  SWA  has  competencies  in  measurement,  data  collection,  analytics,  data  modeling,  systematic 
reviews,  validation,  and  evaluation. 

For  more  information  about  SWA,  our  projects,  and  our  capabilities,  please  visit  our  website  (www.swa- 
consulting.com)  or  contact  Dr.  Eric  A.  Surface  (esurface@swa-consulting.com)  or  Dr.  Stephen  J.  Ward 
( sward@swa-consulting.com) . 
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APPENDIX  A:  ABOUT  THE  LCNA  PROJECT 

In  2003-2004,  the  Special  Operations  Forces  Language  Office  (SOFLO)  sponsored  the  SOF  Language 
Transformation  Strategy  Needs  Assessment  Project  to  inform  the  development  of  a  language 
transformation  strategy  in  response  to  a  GAO  report  (2003).  This  SOF  Language  Transformation  Strategy > 
Needs  Assessment  Project  collected  current-state  information  about  language  usage,  proficiency,  training, 
and  policy  issues  (e.g.,  Foreign  Language  Proficiency  Pay,  FLPP)  from  SOF  personnel,  SOF  unit  leaders, 
and  other  personnel  involved  in  SOF  language.  The  project  used  multiple  data  collection  methods  and 
provided  the  SOFLO  with  valid  data  to  develop  a  comprehensive  language  transformation  strategy  and 
advocate  for  the  SOF  perspective  on  language  issues  within  the  DoD  community. 

In  a  continuing  effort  to  update  knowledge  of  language  and  culture  needs  while  informing  strategic  plan 
development,  the  SOFLO  commissioned  the  2009  SOF  Language  and  Culture  Needs  Assessment  Project 
(LCNA)  to  reassess  the  language  and  culture  landscape  across  the  United  States  Special  Operations 
Command  (USSOCOM)  and  develop  a  strategy  for  the  next  five  years.  Data  were  collected  between 
March  and  November,  2009  from  personnel  in  the  SOF  community,  including  operators  and  leaders. 
Twenty-three  focus  groups  were  conducted  between  March  and  June,  2009.  A  comprehensive,  web-based 
survey  for  SOF  operators  and  leaders  was  launched  on  26  October  and  closed  on  24  November,  2009. 

This  project’s  findings  will  be  disseminated  through  reports  and  briefings  (see  Appendix  B,  Figure  1). 
Two  foundational  reports  document  the  methodology  and  participants  associated  with  this  project.  The 
remaining  reports  are  organized  in  three  tiers.  Twenty-five  Tier  I  reports  focus  on  specific,  limited  issues 
(e.g.,  Inside  AOR  Use  of  Language).  Tier  II  reports  integrate  and  present  the  most  important  findings 
across  related  Tier  /  reports  (e.g.,  Use  of  Language  and  Culture  on  Deployment)  while  including 
additional  data  and  analysis  on  the  topic.  Most,  but  not  all,  Tier  I  reports  will  roll  into  Tier  II  reports.  One 
Tier  III  report  presents  the  most  important  findings,  implications,  and  recommendations  across  all  topics 
explored  in  this  project.  The  remaining  Tier  HI  reports  present  findings  for  specific  SOF  organizations 
[e.g.,  Air  Force  Special  Operations  Command  (AFSOC),  Special  Forces  (SF)  Command].  All  Tier  III 
reports  are  associated  with  a  briefing.  Report  topics  are  determined  by  the  SOFLO  and  subject  to  change. 

In  June,  2009,  the  GAO  reported  that  the  Department  of  Defense  is  making  progress  toward  transforming 
language  and  regional  proficiency  capabilities  but  still  does  not  have  a  strategic  plan  in  place  to  continue 
development  that  includes  actionable  goals  and  objectives.  The  findings  from  this  study  can  be  used  by 
the  SOFLO  and  leaders  at  USSOCOM  to  continue  strategic  planning  and  development  in  this  area. 

This  project  design,  logistics,  data  collection,  initial  analysis  and  first  eight  reports  of  this  project  were 
conducted  by  SWA  Consulting  Inc.  (SWA)  under  a  subcontract  with  SRC  (SR20080668  (K142);  Prime  # 
N65236-08-D-6805).  The  additional  reports  are  funded  under  a  separate  contracting  vehicle  with  Gemini 
Industries  Inc.  [GEM02-ALMBOS-0018  (10210SWA-1);  Prime  #  USZA22-02-D-0015],  For  questions  or 
more  information  about  the  SOFLO  and  this  project,  please  contact  Mr.  Jack  Donnelly 
(iohn.donnelly@socom.mil).  For  specific  questions  related  to  data  collection  or  reports  associated  with 
this  project,  please  contact  Dr.  Eric  A.  Surface  (esurface@swa-consulting.com)  or  Dr.  Reanna  Poncheri 
Harman  (rpharman@swa-consulting.com)  with  SWA  Consulting  Inc. 
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Appendix  A,  Figure  1.  Report  Overview 


Foundation  Reports  Tier  I  Reports  First  Contract 


1.  Methodology  Report  3.  Admiral  Olson's  Memo 

2.  Participation  Report  4.  Training  Emphasis:  Language  and  Culture 

5.  Command  Support:  Grade  Chain  of  Command 

6.  SOFLO  Support 

7.  Inside/Outside  AOR  Use  of  Cultural  Knowledge 

8.  Team  Composition 


Tier  1  Reports  Second  Contract 


9.  Inside  AOR  Use  of  Language 

10.  Outside  AOR  Use  of  Language 

11.  Mission-Specific  Use  of  Interpreters 

12.  General  Use  of  Interpreters 


Tier  II  Reports  Second  Contract 


28.  Use  of  Language  and  Culture  on  Deployment 

29.  Use  of  Interpreters 

30.  Team  Composition  and  Capability 

31.  Testing/Metrics 

32.  Current  State  of  Language  Training 

33.  Language  Training  Guidance 

34.  Culture  Training  Guidance 

35.  Incentives/Barriers 

[ 

\ _ 


Tier  III  Reports  Second  Contract 


13. 09L 

14.  DLPT 

15.  OPI 

16.  DLAB:  Perspectives  from  the  Field 

17.  Initial  Acquisition  Training 

18.  Sustainment/Enhancement  Training 

19.  Culture  Training 

20.  Immersion 


36.  Overall  Picture:  Conclusions  and 
Recommendations 

37. AFSOC 

38.  MARSOC 

39.  WARCOM 

40.  SF  Command 

41.  CA 

42.  PSYOP 


21.  Language  Resources,  Technology  &  Self-Study  43  Seminar  Briefing(s) 

22.  Foreign  Language  Proficiency  Bonus 

23.  Non-monetary  Incentives 

24.  Command  Support:  Other 
Barriers/Organizational  Support 

25.  Force  Motivation  for  Language 

26.  Leader  Perspectives  on  Language  Issues 

27.  CLPM  Perspectives 

Note :  Foundation  reports  are  referenced  by  every  other  report.  Colors  represent  Tier  /reports  that  roll  (integrate)  into  an  associated  Tier  II  report.  Reports  in  black  are  final  reports  on  the  topic  but 
may  be  cited  by  other  reports.  Tier  II  reports  roll  into  the  Tier  III  reports.  All  Tier  III  reports  include  an  associated  briefing. 
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APPENDIX  B:  METHODOLOGY 


Participants 

Focus  Group  Participants 

Twenty-three  focus  groups  were  conducted  with  126  SOF  personnel  across  the  SOF  community.  Focus 
groups  were  conducted  with  Air  Force  Special  Operations  Command  (AFSOC),  Marine  Corps  Forces 
Special  Operations  Command  (MARSOC),  Naval  Special  Warfare  Command  (WARCOM),  and  United 
States  Army  Special  Operations  Command  (USASOC;  see  Participation  Report,  Technical  Report 
#2010011003  for  participant  details).  Section  IV  of  this  report  presents  focus  group  discussion  related  to 
perceptions  of  the  DLAB  (see  Methodology >  Report,  Technical  Report  #2010011002  for  the  focus  group 
interview  guide). 

Survey  Participants 

Survey  respondents  received  the  operator  version  of  the  DLAB  items  if  they  indicated  one  of  the 
following  SOF  community  roles: 

•  SOF  Operator 

•  SOF  Operator  assigned  to  other  duty 

•  Currently  in  the  training  pipeline 

•  MI  Linguist  or  09L  assigned  or  attached  to  a  SOF  unit 

The  focus  of  this  report  is  on  SOF  operator  and  SOF  leader  perspectives,  therefore,  MI  Linguist/09L 
perspectives  are  not  included  in  this  report. 

Survey  respondents  received  the  leader  version  of  the  DLAB  items  if  they  indicated  one  of  the  following 
SOF  community  roles: 

•  SOF  Unit  Commander 

•  Command  Language  Program  Manager  (CLPM) 

•  Language  office  personnel 

•  Other  civilian 

The  focus  of  this  report  is  on  SOF  operator  and  leader  perspectives,  therefore,  CLPM,  language  office 
personnel,  and  civilian  perspectives  are  not  included  in  this  report.  For  further  details  on  participation  and 
attrition  rates,  please  refer  to  the  Participation  Report  (Technical  Report  #2010011003). 

Measures 

Survey  respondents  were  given  two  closed-ended  items  and  three  open-ended  items  related  to  the  DLAB. 
Whether  operators  and  leaders  were  presented  these  items  depended  on  their  response  to  the  following 
items: 

Operators 
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•  Have  you  taken  the  Defense  Language  Aptitude  Battery  (DLAB)? 

Overall,  1,140  SOF  operators  (SOF  operators  or  SOF  operators  assigned  to  other  duty)  responded  to  this 
item.  Eighty-one  percent  ( n  =  920)  of  operators  responded  “Yes”  to  this  item  and  so  they  received  the 
DLAB  follow  up  items.  Operators  that  responded  “No”  were  branched  to  the  next  survey  section. 

Leaders 

•  Are  operators  in  your  unit  required  to  take  the  DLAB?  (If  yes,  are  you  in  a  position  to  comment 
on  the  DLAB?) 

Overall,  808  SOF  leaders  responded  to  this  item.  Eleven-percent  ( n  =  92)  of  leaders  responded  “Yes,  and 
I  am  in  a  position  to  comment  on  the  DLAB”  and,  therefore,  received  the  DLAB  items.  Leaders  that 
responded  “Yes,  but  I  am  not  in  a  position  to  comment  on  the  DLAB”  (46%,  n  =  374),  “No,  operators  in 
my  unit  are  not  required  to  take  the  DLAB”  (24%,  n  =  194),  or  “Not  applicable”  (18%,  n  =  148)  were 
branched  to  the  next  survey  section. 

Follow-up  DLAB  items 

Operators  and  leaders  who  continued  to  the  DLAB  items  received  the  following  closed-ended  items. 
Response  options  included  “Yes”  and  “No”  for  each  item. 

•  Do  you  think  your  DLAB  score  accurately  reflects  [your]2  language  learning  aptitude? 

•  Do  you  think  that  the  DLAB  should  be  used  to  assign  individuals  to  language  training? 

Operators  and  leaders  also  received  three  open-ended  items: 

•  Please  explain  why  the  DLAB  does  or  does  not  accurately  reflect  [your]  language  learning 
aptitude. 

•  Please  explain  why  the  DLAB  should  or  should  not  be  used  to  assign  individuals  to  language 
training. 

•  Use  the  space  below  to  provide  any  specific  feedback  you  have  related  to  the  DLAB. 

Overall,  at  least  32%  of  the  8903  operators  and  58%  of  the  92  leaders  who  responded  to  the  DLAB  survey 
items  provided  at  least  one  comment  in  the  open-ended  item  fields. 

•  Please  explain  why  the  DLAB  does  or  does  not  accurately  reflect  [your]  language  learning 
aptitude. 

o  Operator  response  rate  =  32%  (287  of  890) 
o  Leader  response  rate  =  58%  (53  of  92) 

•  Please  explain  why  the  DLAB  should  or  should  not  be  used  to  assign  individuals  to  language 
training. 

o  Operator  response  rate  =  30%  (268  of  890) 
o  Leader  response  rate  =  53%  (49  of  92) 


2  Words  in  brackets  reflect  item  wording  on  operator  version  of  the  survey. 

3  920  operators  (SOF  operators  and  SOF  operators  assigned  to  other  duty)  responded  ‘Yes’  to  the  first  DLAB  item.  Have  you 
taken  the  Defense  Language  Aptitude  Battety  (DLAB)?  Flowever,  30  of  these  operators  did  not  respond  to  the  remaining  DLAB 
items. 
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•  Use  the  space  below  to  provide  any  specific  feedback  you  have  related  to  the  DLAB. 
o  Operator  response  rate  =  5%  (47  of  890) 
o  Leader  response  rate  =  1 0%  (9  of  92) 


Analyses 

All  closed-ended  items  were  analyzed  using  a  combination  of  descriptive  and  inferential  statistics.  To 
compare  responses  across  groups  of  participants,  inferential  statistics  (e.g.,  chi  square  tests)  were  used  to 
determine  if  any  observed  differences  are  likely  to  exist  in  the  broader  population  of  interest.  Among  the 
groups  compared  included: 

•  Deployment  history 

•  Language-coded  position  vs.  not  in  a  language-coded  position 

•  Official/required  language  difficulty 

•  Proficiency  in  official/required  language 

•  Grade  (within  E,  WO,  and  O) 

To  analyze  the  focus  group  data  and  open-ended  items  (survey  comments),  two  raters  created  a  content 
code  (i.e.,  theme)  list  based  on  available  responses  (see  Methodology >  Report,  Technical  Report 
#2010011002  for  details  on  qualitative  coding).  A  primary  rater  then  coded  each  response  and  a 
secondary  rater  coded  30%  of  the  responses.  Raters  determined  the  consistency  of  codes  applied  between 
raters  and  discussed  any  disagreements  to  consensus.  The  frequency  of  occurrence  for  each  theme  is 
presented  in  this  report. 

For  further  details  on  these  methods  please  refer  to  the  Methodology >  Report  (Technical  Report 

#2010011002). 
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APPENDIX  C:  COMMENT  CODE  DEFINITIONS 

SOF  operators  and  SOF  leaders  were  given  the  opportunity  to  provide  comments  in  response  to  the 
following  survey  prompts: 

•  Please  explain  why  the  DLAB  does  or  does  not  accurately  reflect  [your] 4  language  learning 
aptitude. 

•  Please  explain  why  the  DLAB  should  or  should  not  be  used  to  assign  individuals  to  language 
training. 

•  Use  the  space  below  to  provide  any  specific  feedback  you  have  related  to  the  DLAB. 

All  survey  comments  were  content  analyzed  and  common  themes  extracted.  The  resulting  themes  are 
provided  below,  with  a  definition  of  each  theme  and  verbatim  exemplar  comments  that  illustrate  the 
theme.  For  more  information  about  this  study’s  content  analysis  process,  please  refer  to  the  LCNA 
Methodology’  Report  (Technical  Report  #  2010011002).  Focus  group  comments  were  analyzed  using 
different  themes;  please  refer  to  the  Methodology ?  Report  for  more  information. 

Note:  Exemplar  comments  are  presented  verbatim  and  are  uncorrected  for  spelling  and  other  mistakes. 

DLAB  is  accurate 

•  Received  low  DLAB  score  and  performed  poorly  in  language 

o  Definition:  Respondent  references  a  personal  or  general  occurrence  of  an  individual  who 
scores  low  on  the  DLAB  and  performs  poorly  in  their  assigned  language. 

■  “I  scored  low  which  put  me  in  a  CAT  one  language.  I  have  diffiuclty  with  the 
language  so  I  believe  the  score  was  accurate.” 

■  “I  scored  poorly  on  it  and  have  struggled  considerably  learning  a  new  language.” 

■  “I  did  not  score  all  that  well  and  was  still  assigned  Arabic.  I  struggled  while 
learning  Arabic.” 

•  Received  high  DLAB  scores  and  performed  well  in  language 

o  Definition:  Respondent  references  a  personal  or  general  occurrence  of  an  individual  who 
scores  high  on  the  DLAB  and  performs  well  in  their  assigned  language. 

■  “I  only  know  that  in  my  own  experience  I  tested  well  on  the  DLAB  and  tested 
better  than  all  but  one  of  my  classmates  after  the  language  training  received  at 
SWC  and  can't  think  of  anyone  I've  talked  to  that  scored  low  on  a  DLAB  yet  was 
able  to  gain  a  high  proficiency  in  a  language  after  4-6  months  of  training.” 

■  “I  scored  an  108  on  my  DLAB.  Because  of  this,  I  was  chosen  to  learn  Modem 
Standard  Arabic.  1  passed  all  tests  during  the  language  training  phase  on  the  first 
attempt.  I  like  to  believe  the  DLAB's  level  of  accuracy  is  a  honest  assessment  of 
ones  ability  to  leam  a  new  language.” 

•  General  comment  that  DLAB  accurately  measures  language  learning  aptitude 

o  Definition:  Respondent  says  that  DLAB  is  an  accurate  measure  of  language  learning 
aptitude  without  referencing  personal  or  general  occurrence  of  low  scoring  individual 
performing  poorly/high  scoring  individual  performing  well. 


4  Words  in  brackets  reflects  item  wording  on  operator  version  of  the  survey. 
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■  “I  believe  the  test  accurately  reflects  the  students  aptitude  for  learning 
languages.” 

■  “I  think  that  the  DLAB  accurately  measures  one's  ability  to  think  in  the 
conceputal  style  that  is  required  for  learning  a  language.  The  way  one  learns 
languages  can  be  quantified  and  measured  by  the  DLAB.” 

■  “The  rate  which  I  learned  a  language  was  what  I  expected  after  taking  the 
DLAB.” 


DLAB  is  inaccurate 

•  DLAB  doesn’t  account  for  motivation/attitude/effort 

o  Definition:  Respondent  says  that  the  DLAB  does  not  account  for  an  individual’s 
motivation  to  learn  a  language. 

■  “It  doesn't  measure  effort  or  interest,  the  two  biggest  factors  in  success.” 

■  “It  has  no  way  to  accuratly  apply  a  persons  "want  to  learn".  If  a  person  wants  to 
learn  a  specific  language  they  will  naturally  try  harder  and  do  better  than  whan 
they  score  on  the  DLAB.” 

■  “Learning  a  foreign  language  really  depends  on  the  Soldiers  motivation  to  learn 
the  language.  A  Soldier  can  get  a  high  score  on  the  DLAB  but  have  no  interest  in 
the  language  they  are  given  to  learn.  If  a  Soldeir  has  no  motivation  to  learn  a 
language  they  are  not  going  to  retain  it.” 

■  “DLAB  scores  do  not  accurately  reflect  language  apitutde  because  motivation  to 
learn  is  not  measured.” 

•  Received  low  DLAB  score  and  does  well  in  difficult  language 

o  Definition:  Respondent  references  a  personal  or  general  occurrence  of  an  individual  who 
scored  low  on  the  DLAB  and  performed  well  in  their  assigned  language. 

■  “I  learned  English  after  learning  to  speak  2  other  languages.  I've  taken  the  DLAB 
it  showed  that  I  don't  have  the  ability  to  learn  a  language,  how  can  that  be 
possible  when  I  speak  close  to  4  languages.” 

■  “According  to  my  DLAB  score  I  was  supposed  to  be  able  to  learn  only  a  CAT  1 
language.  I  was  trained  in  a  CAT  3  language  and  passed  the  DLPT  with  a  1/1.” 

■  “I  had  one  of  the  lowest  scores  from  my  DLAB  Test.  I  still  got  ARABIC  and  was 
on  the  Commadants  List.  An  I'm  the  second  best  speaker  on  my  Detachment.  The 
best  is  a  former  MI  guy  who  swithed  to  SF.” 

■  “Due  to  the  fact  I  scored  low  on  the  DLAB  and  still  qualified  to  speak  Korean, 
which  is  a  harder  language  than  I  should  have  been  able  to  learn” 

•  Received  high  DLAB  score  and  does  poorly  in  difficult  languages 

o  Definition:  Respondent  references  a  personal  or  general  occurrence  of  an  individual  who 
scored  high  on  the  DLAB  and  performed  poorly  in  their  assigned  language. 

■  “I  went  to  DLI  with  a  student  who  had  a  DLAB  score  of  135.  He  failed  out  of  the 
class  because  of  his  inability  to  grasp  the  language.  Our  honor  graduate  had  a 
DLAB  score  of  107.” 

■  “Scored  high,  had  difficult  time  learning” 
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•  “I  scored  high  on  DLAB  but  the  harder  languages  are  difficult  for  me  to  do  more  than  memorize 
phrases. ’’Other  factors  that  DLAB  doesn’t  account  for  (e.g.,  study  habits,  learning  strategies) 

o  Definition:  Respondent  says  that  the  DLAB  doesn’t  account  for  additional  individual 
differences  or  situational  factors  excluding  motivation/attitude/effort. 

■  “Different  learning  styles  may  prevent  students  from  being  as  successful  during 
DLAB  as  they  are  during  DLPT  based  on  self  study  and  motivation  during  the 
course  work.” 

■  “it  doesn't  gauge  that  some  people  learn  language  in  different  ways,  and  that  also 
a  persons  interpersonal  skills  can  make  them  much  more  effective,  basically  if 
your  well  liked  the  locals  will  also  try  harder  to  understand  you  and  work  with 
you” 

■  “Tests  ans  mesurments  are  relative,  and  do  not  assess  previous  experiance.” 

•  Some  people  aren’t  good  at  taking  tests 

o  Definition:  Respondent  says  that  the  DLAB  measures  test-taking  ability  or  that  the 

DLAB  is  not  an  accurate  measure  of  language  learning  aptitude  because  it  measures  test¬ 
taking  ability. 

■  “Some  people  simply  do  not  take  tests  well.  Their  results  do  not  accurately 
reflect  their  aptitude.” 

■  “It  did  not  make  any  sense.  Some  people  just  don't  test  well.  That  doesn't  mean 
they  can  not  learn.” 

■  “Again  this  measure  you  ability  to  take  a  test.  College  grads  do  well  and  high 
school  grads  are  lacking.  The  high  school  grads  simply  don't  have  the  experience 
in  takin  test.” 

•  DLAB  doesn’t  test  all  aspects  of  language  learning  aptitude  (e.g.,  does  not  test  one’s 
understanding  of  English) 

o  Definition:  Respondent  says  that  the  DLAB  doesn’t  test  all  mechanical  aspects  of 
language  acquisition. 

■  “It  cna  help  indicate  who  cna  learn  a  language  but  Soldier's  knowledge  of 
English  needs  to  be  assessed  as  well.  If  one  does  not  know  what  a  past  participle 
is  in  English  how  can  one  learn  it  in  another  language?” 

■  “the  DLAB  reflects  potential-  but  when  you  actually  have  to  learn  a  real 
language  you  must  know  English  rules  first-  not  some  made-up  rules  you  learn 
for  a  few  minutes.  If  you  don't  know  English  rules  and  the  presentation  is  done 
using  English  rules  to  learn  the  new  language  rules-  then  it  is  hard. ...” 

•  DLAB  not  based  on  a  real  language 

o  Definition:  Respondent  says  that  the  DLAB  is  not  based  on  a  real  language  or  that  the 
DLAB  is  not  an  accurate  measure  of  language  learning  aptitude  because  it  is  not  based  on 
a  real  language. 

■  “How  can  you  assess  ones  ability  to  learn  a  language  with  some  crazy  made  up 
language?  I  know  people  that  are  native  speakers  in  a  number  of  different 
languages  and  they  came  out  of  the  DLAB  with  the  same  headache  and 
confussion  I  had.” 
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•  DLAB  was  more  difficult  than  learning  the  language 

o  Definition:  Respondent  says  that  the  DLAB  was  more  difficult  than  learning  their 
assigned  language. 

■  “I  found  the  DLAB  far  more  difficult  to  understand  than  the  target  Language, 
Farsi  in  this  case.” 

•  Testing  condition  issues  with  DLAB 

o  Definition:  Respondents  say  that  they  had  test  condition  issues  while  taking  the  DLAB. 

■  “The  test  was  given  under  stressful  conditions.” 

■  “During  the  course  of  my  DLAB  I  was  unaware  of  a  final  section  of  the  exam 
until  very  little  time  remained.  As  a  result  I  was  unable  to  adequately  attend  to 
the  questions  in  this  section  and  my  score  consequently  suffered.” 

■  “I  fell  asleep  when  I  took  the  DLAB  because  or  inadquate  facilities  (the  room 
was  100  degrees  in  the  middle  of  a  hot  North  Carolina  summer)  and  because  our 
drill  sergeants  did  not  let  us  get  enough  sleep  the  night  before  and  the  took  us  on 
a  8  mile  run  right  before  taking  the  test.  The  whole  situation  set  everyone  who 
took  that  test  up  for  failure.” 

•  General  comment  that  DLAB  inaccurately  measures/fails  to  measure  language  learning  aptitude 

o  Definition:  Respondents  say  that  the  DLAB  does  not  accurately  measure  or  fails  to 
measure  language  learning  aptitude,  without  referencing  personal  or  general  occurrence 
of  low  scoring  individual  performing  well/high  scoring  individual  performing  poorly. 

■  “The  DLAB  is  more  or  less  based  on  guessing.” 

■  “I  scored  a  135  on  my  DLAB  compared  to  fellow  soldiers  in  my  class  that  scored 
95  or  lower,  yet  achieved  the  same  score  (2  listening,  2  reading)  on  the  DLPT 
after  language  school.” 

■  “I  do  not  see  the  corelation  between  the  score  and  my  ability  to  learn  a 
language;” 

Other  comments  about  DLAB 

•  Perceptions  of  what  the  DLAB  tests 

o  Definition:  Respondent  describes  what  he  or  she  thinks  the  DLAB  tests. 

■  “Understanding  the  rules  of  a  language  in  a  short  amount  of  time” 

■  “It  shows  the  ability  to  recognize  patttems  and  rules  associated  with  languages.” 

■  “i  think  it  just  tests  a  persons  capability  to  understand  some  language  rules  (usage 
of  verbs/nouons/etc” 

■  “the  DLAB  is  a  simple  language  based  IQ  test.” 

■  “IT  GIVES  YOU  A  MEASURE  OF  AN  INDIVIDUALS  ABILITY  TO  LEARN 
A  PARTICULAR  LANGUAGE.” 

•  DLAB  is  the  best/only  available  language  aptitude  test 

o  Definition:  Respondent  says  that  the  DLAB  is  the  best  available  or  the  only  available 
language  aptitude  test. 

■  “i  guess  it  is  as  good  a  tool  as  any,  i  am  not  aware  of  any  other  test  that  measure 
aptitude.” 
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■  “You  have  to  use  some  tool.  It  is  better  than  shooting  in  the  dark.” 

■  “Something  is  better  than  nothing,  and  right  now  the  DLAB  is  it.” 

■  “No  other  way  to  measure  language  aptitude.” 

•  Lack  of  knowledge  about  DLAB 

o  Definition:  Respondent  is  confused  or  unsure  about  what  the  DLAB  tests  or  how  the 
DLAB  scores  are  used  to  assign  individuals  to  languages. 

■  “I  am  uncertain  as  to  just  what  the  DLAB  measures.  No  explanation  has  been 
provided  to  explain  how  the  DLAB  relates  to  a  person's  language  apptitude.” 

■  “I  have  no  relevent  opinion  on  this  question.  I  have  no  idea  how  they  use  the 
results  to  assign  personnel.” 

■  “I  don't  understand  how  the  DLAB  works.” 

•  DLAB  scores  aren’t  used  to  assign  people  to  languages 

o  Definition:  Respondent  says  that  DLAB  scores  are  not  used  to  assign  individuals  to 
language  within  the  SOF  community. 

■  “The  DLAB  had  nothing  to  do  with  language  assignment  when  I  went  through 
the  language  course.” 

■  “It  should  be  used  but  I  got  like  a  73  and  they  gave  me  arabic  so  they  don't  use 
the  DLAB  for  anything.” 

■  “I  did  not  score  high,  and  still  got  a  cat4  language. . .” 

•  Suggestions  regarding  language  assignments 

o  Definition:  Respondent  suggests  revision  to  existing  language  assignment  process. 

■  “Many  individuals  have  only  a  limited  ability  to  learn  a  language.  Those 
individuals  should  be  required  to  basic  memorization  or  the  target  language  and 
practice  using  language  tools  and  interpreters.” 

■  “Blanket  SF  students  with  a  basic  language  school.  Let  the  Groups  focus  on 
language  training  improvements  by  selecting  those  individuals  with  the  desire 
and  ability  to  increase  their  capability.” 

■  “This  is  only  a  gage.  The  language  selection  of  the  SM  should  match  the  AOR 
regardless  of  DLAB  scores.” 

■  “Provide  the  individual  with  the  DLAB  study  guide  (2  page  guide).  It  increased 
my  score  45  points.” 

•  General  positive  DLAB  comment 

o  Definition:  Respondent  provides  a  general  positive  comment  about  the  DLAB. 

■  “I  would  rather  answer  the  above  question  with  a  "neither  agree  nor  disagree".  I 
think  the  DLAB  will  help  prevent  personnel  with  no  aptitude  for  more  difficult 
languages  from  having  to  suffer  through  a  CAT  4  language  unecessarily.  I  may 
be  a  fluke  but  I  think  there  is  some  merit  to  the  test.” 

■  “It  at  least  provides  some  sort  of  basis  to  start  assesing  ones  aptitude  at 
language.” 

•  General  negative  DLAB  comment 

o  Definition:  Respondent  provides  a  general  negative  comment  about  the  DLAB. 
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Defense  Language  Aptitude  Battery  (DLAB) 


■  “DLAB  should  only  be  one  of  several  inidcators  of  possible  aptitude.  It  is  a 
VERY  POOR  stand-alone  indicator.” 

■  “Its  an  ineffective  test” 

■  “The  placement  of  soldiers  by  using  the  DLAB  is  not  the  best  manner  in  which  to 
do  so.” 

•  Not  relevant  to  the  DLAB 

o  Definition:  Respondent  provides  a  comment  that  is  unrelated  to  the  DLAB. 
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